It is correct that dates do not fit nicely into the Stevens typology https://en.wikipedia.org/wiki/Level_of_measurement#Ordinal_scale of different levels of measurement. Dates are certainly ordered, so we could say that dates are ordinal type, but they are certainly more than that. When talking specifically about days in this sense, astronomers use Julian days.
I take your question to be what mathematical structure can we give to the set of dates (or more generally dates/times). That is about a mathematical representation of time, and we talk generally of time in at least two ways: events ("when did something happen") and durations "how long did the last winter Olympic games in PyeongChang last"? If $P$ is the date of the opening ceremony and $Q$ the date of the closing ceremony, then the duration is $Q-P$. So we can take a difference of two events (dates); that difference is a duration. But we cannot sum two events (dates), what should we mean by $P+Q$? But the halfway point of the winter Olympics has meaning; that is the average $0.5 P+0.5 Q$. So averages make sense!
This looks like a strange mathematical structure, with two kinds of objects "events" and "durations" and operations only defined in some cases, not all. But it is a very well-known object, an affine space; see https://en.wikipedia.org/wiki/Affine_space.
The usual way of introducing an affine space is saying it is a vector space "where we have forgotten the origin". Since we have forgotten the origin, any operation whose result depends on the origin is invalid or undefined. We can now define "events" (dates) as vectors in the underlying (1-dim) vector space, which we can identify with the real line. But note that this representation depends on choice of an origin! We must just remember that anything we actually do must not depend on this choice.
We can represent "durations" as differences between the vectors representing dates. It should be quite obvious that the duration of the winter Olympic Games do not depend on if we choose as time origin the birth of Christ or 1 january 1970 (time origin used in linux). The average of events also has meaning: if we write the events as $P_i$, then the average of the $P_i$ is an event $Q$ such that
$$
\sum_i (P_i - Q)=0
$$ (In affine geometry $Q$ is called often the barycenter.)
Note that here we are only summing durations, which is allowed.
If we want to implement some data type representing dates in a computing environment, it must have these properties. Let us see in R:
P <- as.Date("2018-2-9") # Starting date of Olympics
Q <- as.Date("2018-2-25") # end date
Q-P # duration
Time difference of 16 days
Q+P
Error in `+.Date`(Q, P) : binary + is not defined for "Date" objects
mean(c(P, Q)) # time midpoint of the games
[1] "2018-02-17"
weighted.mean(c(P, Q), c(1/4, 3/4)) # games 3/4-finnished.
[1] "2018-02-21"
P+16 # 16 days after the opening ceremony
[1] "2018-02-25"
That all seems to be well-behaved.
