The coefficient of determination is $$r^2 = 1 - \frac{SS_\text{res}}{SS_\text{tot}}$$ where $SS_\text{res}=\sum_{i=1}^n (y_i-\hat{y_i})^2$ and $SS_\text{tot}=\sum_{i=1}^n (y_i - \overline{y})^2$.
Why is this used for assessing the fit of a least squares line? Why is the comparison based on looking at $SS_\text{res}/SS_\text{tot}$ as opposed to say $SS_\text{res}/n$ or something else?
This is a very broad question, although it may not seem so. Two comments:
You say "The coefficient of determination is" but whether the formula you give acts as a definition of fundamentals for anyone is unclear. I'd characterise it rather as one of several available computing formulas.
You ask "Why is this used" but that confuses or conflates the question of why the coefficient of determination is used at all with why the particular formula you cite might be used.
For me, the attractions of $R^2$ lie in being (a) a simple and single measure linked to the correlation coefficient $r$ or an analogue of that and (b) free of the units of measurement of the original variable. In multiple regression, the correlation concerned is between the values observed and those predicted from the model.
The disadvantages of $R^2$ are precisely the same points: no summary measure can capture all the virtues and limitations of a regression and there is often much point in summarising lack of fit on the scale of the measured response.
To that end, $SS_\text{res}/n$ is, contrary to your implication, often used, if indirectly. Summarising the residuals by mean square is at base a good idea, although its square root is a better one on dimensional grounds and for detailed technical reasons there is a case for using a divisor which is the sample size minus the number of parameters fitted. (Looking at the detailed pattern of the residuals is usually an even better idea.)
More broadly, $R^2$ is often over-valued in that a low $R^2$ may be a worthwhile achievement and a high $R^2$ a scientific or practical failure. Much depends on what is interesting, useful and possible scientifically or practically.
The $SS$ can be considered a sum quantity of variability. The $SS_\text{tot}$ is all of the variability when the very simplest model is used, the mean. Look at the equation, it's the sum of each squared deviation, all of that variability not explained by the mean (any value exactly at the mean contributes 0 to $SS$). The $SS_\text{res}$ is the variability that your more complex model didn't explain, whatever that model is. For example, if you have two means in the more complex model they should explain more of the data / have a smaller $SS$. Therefore $SS_\text{res}/SS_\text{tot}$ is the proportion of variability that you didn't explain. If you subtract what's unexplained from 1 then you get the remaining portion of variability you did explain.
It means something. The reason it's used is because it means something sensible and useful. $SS_\text{res}/n$, or some other value, may mean something too, but not the same thing. If you come up with a more useful number for your purposes then use that.
