A student-$t$ confidence interval is quite robust to deviations from normality. If the data is uniformly distributed, the following simulation shows that the student-$t$ interval is slightly anti-conservative with a true confidence level around 0.947, for a nominal level of 0.95 and a sample size of $n=10$.
> a <- 0
> b <- 1
> mu <- (a+b)/2
> nsim <- 1e+5
> hits <- 0
> n <- 10
> for (i in 1:nsim) {
+ x <- runif(n, a, b)
+ ci <- confint(lm(x ~ 1))
+ if (ci[1] < mu & ci[2] > mu)
+ hits <- hits + 1
+ }
> hits/nsim
[1] 0.94738
In terms of coverage, you're thus not doing any big mistake by relying on this interval.
You can get a mostly shorter interval with exact coverage, however, by constructing the interval from the smallest and largest observations $X_{(1)}$ and $X_{(n)}$. These are sufficient statistics for $a$ and $b$. It follows that
$$
Z_1=\frac{\frac{X_{(1)} + X_{(n)}}2-\frac{a+b}2}{X_{(n)}-X_{(1)}} \tag{1}
$$
is a pivotal quantity since the distribution of $Z_1$ doesn't depend on $a$ and $b$, only $n$. This follows because we alternatively can write (1) as
$$
Z_1=\frac{U_{(1)} + U_{(n)}-1}{2(U_{(n)}-U_{(1)})} \tag{2}
$$
where $U_{(1)}$ and $U_{(n)}$ denote the corresponding order statistics associated with $n$ observations from a uniform distribution on (0,1).
The joint density of $U_{(1)}$ and $U_{(n)}$ is
$$
f_{U_{(1)},U_{(n)}}(u_1,u_n)=n(n-1)(u_n-u_1)^{n-2}.
$$
for $0<u_1<u_n<1$.
Letting
$$
Z_2 = U_{(n)}-U_{(1)},
$$
the joint density of $Z_1,Z_2$ is
\begin{align}
f_{Z_1,Z_2}(z_1,z_2)
&=f_{U_{(1)},U_{(n)}}(u_1(z_1,z_2),u_n(z_1,z_2))\left|\begin{matrix}\frac{\partial u_1}{\partial z_1} & \frac{\partial u_1}{\partial z_2} \\ \frac{\partial u_n}{\partial z_1} & \frac{\partial u_n}{\partial z_2}\end{matrix}\right|
\\&=n(n-1)z_2^{n-2}\left|\begin{matrix}z_2 & z_1 -\frac12 \\ z_2 & z_1 +\frac12\end{matrix}\right|
\\&=n(n-1)z_2^{n-1},
\end{align}
for $-\infty<z_1<\infty$ and $0<z_2<\frac1{2|z_1|+1}$. Elsewhere the density is zero. Hence, the density of the pivot $Z_1$ is
$$
f_{Z_1}(z_1)=\int_0^{\frac1{2|z_1|+1}}f_{Z_1,Z_2}(z_1,z_2)dz_2=\frac{n-1}{(2|z_1|+1)^n}.
$$
Further calculations shows that the upper $\alpha/2$-quantile of this symmetric distribution is given by
$$
q_{\alpha/2} = \frac12\left( \alpha^{-\frac1{n-1}} - 1 \right).
$$
Inverting the double inequality involving (1), an exact $(1-\alpha)$ confidence interval for the mean $(a+b)/2$ is thus
$$
\frac{X_{(1)}+X_{(n)}}2 \pm \frac{X_{(n)}-X_{(1)}}2\left( \alpha^{-\frac1{n-1}} - 1 \right).
$$
Further simulations shows that the length of the exact interval indeed is considerably shorter:
> l1 <- l2 <- numeric(nsim)
> hits2 <- 0
> n <- 10
> pm <- c(-1,1)
> for (i in 1:nsim) {
+ x <- runif(n, a, b)
+ ci <- confint(lm(x ~ 1))
+ l1[i] <- ci[2]-ci[1]
+ ci2 <- (min(x)+max(x))/2 + pm*(max(x)-min(x))/2*(1/0.05^(1/(n-1))-1)
+ l2[i] <- ci2[2]-ci2[1]
+ if (ci[1] < mu & ci[2] > mu)
+ hits <- hits + 1
+ if (ci2[1] < mu & ci2[2] > mu)
+ hits2 <- hits2 + 1
+ }
> hits2/nsim
[1] 0.94917
> mean(l1)
[1] 0.407362
> mean(l2)
[1] 0.3230234