It seems to be not very useful for ARIMA, while checking the stationary assumption using the residual is more important.
Let me note that this is the same as in other statistical models, e.g. linear regression model, so ARIMA is not special in this respect.
It also does not mean the expected value of $y_t$, since ARIMA assumes a constant mean.
But it is an estimate of the conditional mean of $y_t$ given the past information. See this answer for a convenient representation of ARMA in terms of the conditional mean of $y_t$.
What's the meaning of the fitted value, despite its mathematical definition I give above?
See the previous point. Also, it can be useful when the series has missing values. The fitted value could substitute for a missing value.
What's its relation with $y_{ob,t}$: if $y_{ob,t}$ is the observed value at time $t$ out of the ensemble, what does the fitted value mean? Is it similar to the hidden state (of $y_{ob,t}$) if we write ARIMA into a state-space model?
It depends on how you formulate the state space model; ARIMA models have several different state space representations. I am not sure if any of these treat the conditional mean of $y_t$ as the latent state, but if it does / they do, then you are right. As noted above, the fitted value stands for the estimated conditional mean of the series.
If it has some meanings, how do we normally use it in analysis?
We do not use it that much by itself, but imputation of missing data points is one example where it can be of interest.
