Statistical Analysis Stack Exchange
Statistical Analysis Stack Exchange

Questions tagged [irls]

IRLS stands for Iteratively Re-weighted Least Squares. IRLS is a commonly used method to find maximum likelihood estimates when they cannot be found analytically.

IRLS stands for Iteratively Re-weighted Least Squares. IRLS is a commonly used method to find maximum likelihood estimates when they cannot be found analytically.

40 questions
40
votes
2 answers

Purpose of the link function in generalized linear model

What is the purpose of the link function as a component of the generalized linear model? Why do we need it? Wikipedia states: It can be convenient to match the domain of the link function to the range of the distribution function's mean What's the…
Chris
  • 1,169
  • 3
  • 12
  • 16
26
votes
1 answer

Why using Newton's method for logistic regression optimization is called iterative re-weighted least squares?

Why using Newton's method for logistic regression optimization is called iterative re-weighted least squares? It seems not clear to me because logistic loss and least squares loss are completely different things.
Haitao Du
  • 32,885
  • 17
  • 118
  • 213
19
votes
1 answer

Definition and Convergence of Iteratively Reweighted Least Squares

I've been using iteratively reweighted least squares (IRLS) to minimize functions of the following form, $J(m) = \sum_{i=1}^{N} \rho \left(\left| x_i - m \right|\right)$ where $N$ is the number of instances of $x_i \in \mathbb{R}$, $m \in…
Chris A.
  • 615
  • 5
  • 15
19
votes
1 answer

Can you give a simple intuitive explanation of IRLS method to find the MLE of a GLM?

Background: I'm trying to follow Princeton's review of MLE estimation for GLM. I understand the basics of MLE estimation: likelihood, score, observed and expected Fisher information and the Fisher scoring technique. And I know how to justify simple…
ihadanny
  • 2,596
  • 3
  • 19
  • 31
9
votes
4 answers

Standard algorithms for doing hierarchical linear regression?

Are there standard algorithms (as opposed to programs) for doing hierarchical linear regression? Do people usually just do MCMC or are there more specialized, perhaps partially closed form, algorithms?
John Salvatier
  • 4,032
  • 1
  • 18
  • 28
9
votes
1 answer

How to correctly implement iteratively reweighted least squares algorithm for multiple logistic regression?

I'm confused about the iteratively reweighted least squares algorithm used to solve for logistic regression coefficients as described on page 121 of The Elements of Statistical Learning, 2nd Edition (Hastie, Tibshirani, Friedman 2009). The final…
RobertF
  • 4,380
  • 6
  • 29
  • 46
8
votes
2 answers

What are some reasons iteratively reweighted least squares would not converge when used for logistic regression?

I've been using the glm.fit function in R to fit parameters to a logistic regression model. By default, glm.fit uses iteratively reweighted least squares to fit the parameters. What are some reasons this algorithm would fail to converge, when used…
Jessica
  • 1,781
  • 2
  • 15
  • 17
7
votes
1 answer

If $\operatorname{Var}\left(\epsilon_i\right) = h\left(X\right) \neq \sigma^2$, what can we know about $\operatorname{Var}\left(\hat{\beta}\right)$?

This question uses the derivations found here. The short version Consider a regression model. If the error variance is a known function of the data (rather than a constant), under what conditions can we draw conclusions about the OLS estimates? The…
shadowtalker
  • 11,395
  • 3
  • 49
  • 109
6
votes
1 answer

Strange variance weights for Poisson GLM for square root link

Is there a reason why all the variance weights from a Poisson GLM are equal when a square root link is used? That is (on R): glm(Y~X1+X2+X3+X4, family=poisson(link=sqrt), data=df)$weights Always returns equal weights for all observations when a…
Anna Efron
  • 182
  • 6
5
votes
1 answer

Iteratively Reweighted Least Squares - Weights Confusion

In performing Iteratively Reweighted Least Squares (IRLS) to derive $\hat{\beta}$ estimates for logistic regression, all the resource I've read online say to use weights inversely proportional to the variance of each $Y_i$. For example, see step 4,…
bob
  • 625
  • 3
  • 14
5
votes
0 answers

IRLS for truncated normal GLM

I have data for which responses fall in $y \in [0,\infty)$ for which, it seems, the standard GLMs based on, say, gamma or inverse-Gaussian fail since they don't allow responses with values equal to 0. So, I am trying to put together a GLM based on a…
Confounded
  • 513
  • 2
  • 12
5
votes
1 answer

Heteroscedasticity that depends on the regression parameters

Consider a vector of observations $\mathbf{Y}$ that can be modeled as \begin{equation} \mathbf{Y} \sim \mathcal{N}( \mathbf{H}\boldsymbol{\beta} , \boldsymbol{\Sigma} ) \end{equation} with $\mathbf{H}$ a linear parametric model, and…
userKT
  • 51
  • 2
5
votes
1 answer

Choosing IRLS over gradient descent in logistic regression

I am currently reading Bishop [1] and got confusion on why should we take IRLS (Iterative Re-weighted Least Square) as it seems that using gradient descent that with one derivative at a time would solve the problem, what is the meaning of…
xxx222
  • 397
  • 5
  • 14
4
votes
0 answers

Would this modification accelerate convergence of generalized linear model, or break it?

This page describes the following iteratively reweighted linear least-squares (IRLS) method for solving a generalized linear model (GLM): let $x_1=0$ for $j=1,2,...$ do linear predictor: let $\eta=Ax_j$ Newton-method correction: let…
Museful
  • 365
  • 2
  • 10
4
votes
2 answers

Are GLMs just glorified WLS regressions?

When performing weighted least squares $L = \frac{1}{2} \sum_i w_i r_i^2$, Aitken showed that one ought to weight each sample by the inverse of its variance $w_i=1/\sigma_i^2$. This leads to gradients of the form $$\nabla_\beta L = \sum_i…
Hyperplane
  • 318
  • 1
  • 9
1
2 3