This question directly relates to this question and generally only differs in the fact that I don't want to fit a GLMM but rather a "regular" Linear Mixed Model. The given answer on using the MAD therefore does not apply.
My situation is as follows: In an earlier study, data from two experimental conditions was collected in a within-subjects approach (each participant completed both conditions per session).
Based on this data, a LMM was fitted using lmerTest.
For simplicity reasons, let's suppose the fitted model looked like this:
m_resp <- lmer(resp ~ age * condition + (1|id), data = df)
with age being a by-participant value.
In a second study, data from $n = 60$ new participants was collected. In a regular linear model, I could now predict the $\hat{y}$ for the new data set using the fitted model from the old data set and calculate the RMSE.
In principle, this also works for the LMM (e.g. using predict.merMod from lme4). However, this (of course) produces two predicted values per participant (leading to a total of 120 predicted values). Therefore, calculating the RMSE from these values by plainly averaging over all predicted values (without taking the grouping per participant into consideration) seems wrong to me. However, I could not find a source with guidance on what/how to do this instead.
This leads me to two questions (the second more on the topic of interpreting the RMSE; might need to go to a different thread as it is more general):
1. Is the RMSE a good/valid assessment criterion in judging the fit for new data in a LMM? If not, which criterion should be used here?
2. How would you actually use the RMSE in such an assessment? Would you compare the RMSE of the new model with the RMSE in the old one? Are there normalized threshold values for good/bad fit?
