I have read "Neural Ordinary Differential Equations" by Chen and coworkers and find it extremely interesting (https://arxiv.org/pdf/1806.07366.pdf).
There is one caveat that I seem to be missing though: In standard ODEs (physics, math etc.) there is always a discretization error no integration scheme can get rid of, which just stems from the fact that there is no 'true' continuity on a computer for processes evolving in time. I.e. there is discretization even when solving ODEs (i.e. another type of discretization compared to ResNets) since you are computing on a time grid after all.
In the figures of the paper (all of them) the computation of the time derivative at a certain point in latent space z is marked by a dot. Until the computation of a new time derivative (i.e. the next dot) this time derivative cannot be adjusted (for lack of better knowledge). Yet the curves of Figure 2 and especially figure 8 seem to be curved even between the computations of new time derivatives. How is this possible? Are they curved due to some interpolation scheme that is not mentioned in the text or am I missing something important?
Thanks to anyone sharing their insight!
