understanding bias correction in ADAM algorithm

Asked Sep 30 '19 at 21:24

Active Oct 01 '19 at 04:37

Viewed 634 times

why is possible to approximate E[gi^2] with E[gt^2]? by the time we go to the t timestamp, we've already made weight updates, which mean gradients should be different as they are taken from different points in parameter space?

I guess we get error term(zeta) by making same approximation but are we really getting rid of that by doing the so-called bias correction, I don't think so

edited Oct 01 '19 at 04:37

asked Sep 30 '19 at 21:24

Kethan Chauhan

I have edited the question, it's not a duplicate one – Kethan Chauhan Oct 01 '19 at 04:39
In that case use the flag button above. – user2974951 Oct 01 '19 at 14:02
@user2974951, there's no need to flag. Editing a closed question bumps it into the reopen queue, where members of the community will decide whether to reopen it. – gung - Reinstate Monica Oct 01 '19 at 18:55

understanding bias correction in ADAM algorithm

0 Answers0