5

I can't be the first person to think about estimating variance components of mixed models by gradient descent, and then computing BLUP's at each update. Googling, I find little on the topic. But the gradients seem tractable (if messy). I suppose one would be logging the variances to render them continuous. I also suppose one would need to penalize the loss function, otherwise the variances would go to infinity.

Can anyone provide any further insight before I go and program it? References? Software? Tips/tricks for optimization? Would minibatch or stochastic gradient descent be problematic?

generic_user
  • 11,981
  • 8
  • 40
  • 63

1 Answers1

1

I would first look into the optimization methods available in R' lme4: See for example Section 4.2 of [1] and the lme4 paper [2].

The penalized least squares algorithm seems to be using gradient information to provide an iterative solution.

[1] https://cran.r-project.org/web/packages/lme4/vignettes/Theory.pdf

[2] https://www.jstatsoft.org/article/view/v067i01/v67i01.pdf

Bar
  • 2,492
  • 3
  • 19
  • 31