4

I discovered a parabolic relationship between time and a quantity in my time series data that looks like the one below: enter image description here

How do I go about building a model that can learn the shape of these parabolas from thousands of samples, and estimate the equation for the curve based on the initial trajectory of a new series (e.g., t=0 to 5).

I am not very technical and I have no idea if this is a reasonable request.

The quantity is viewers per minute of a video stream. Trajectory is metaphorical. At the end of the day I want to be able to input some array of t(3,5,8..n) of viewers for n minutes and get a better and better projection of the curve the more data points I give it, such that I can estimate max height and area under the curve, and do backtesting.

Some resources I found that seem relevant: https://stat.ethz.ch/R-manual/R-devel/library/stats/html/predict.smooth.spline.html and Equation of a fitted smooth spline and its analytical derivative

I can fit a spline to trajectory but I don't know where to go from there:

d.spl <- with(d, smooth.spline(t, viewers))
d.spl

Call: smooth.spline(x = t, y = viewers)

Smoothing Parameter spar= 0.2297785 lambda= 0.0000001153716 (12 iterations) Equivalent Degrees of Freedom (Df): 60.23867 Penalized Criterion (RSS): 10483.72 GCV: 175.7969

enter image description here

metalaureate
  • 93
  • 7
  • 2
    These are not distributions but relationships between your observed quantity and time (which we might metaphorically refer to as *trajectories*) . When you call it a parabola, do you mean that you intend to fit an actual parabola or is it intended more to mean something like "a somewhat-near-to-parabolic more-or-less-smooth function that increases and then decreases"? What's the quantity? Is there some subject-specific underlying theory (as there would be for actual object-trajectories) that would inform the model-choice? – Glen_b Nov 04 '17 at 01:40
  • 1
    The quantity is viewers per minute of a video stream. Trajectory is metaphorical. At the end of the day I want to be able to input some array of t(3,5,8..n) of viewers for n minutes and get a better and better projection of the curve the more data points I give it, such that I can estimate max height and area under the curve, and do backtesting. – metalaureate Nov 04 '17 at 15:03
  • 2
    This information would be useful to have in your post; for example, the fact that the data are counts is likely to be important. – Glen_b Nov 04 '17 at 15:37
  • 1
    I like this problem -- few more clarifying questions (since it's a little hard to tell from your plot). 1) Is there a "natural" END to the trajectories? I assume the views start when the video is posted, but are they taken down? The end looks pretty abrupt, just want to know if I should be thinking about an arbitrarily long tail. 2) You know which viewers are watching which video? – one_observation Nov 06 '17 at 00:49
  • 1
    I see that data on the left side of your image might be modeled as individual parabolas, but data on the right side of the image does not appear parabolic. If you are not interested in the flat-looking data on the right side of the image, your modeling will be more accurate if you filter out data below some minimum value to eliminate the right-side data. – James Phillips Nov 06 '17 at 00:52
  • @Sophologist for 1) these are live video streams, so they generally tend when viewership falls off below a certain threshold ~20 active viewers; they can be taken down also; and the streamer could end them for any reason; 2) yes – metalaureate Nov 06 '17 at 02:24

1 Answers1

2

A quick answer, not robust but a good starting place, which I'll add to should I have time (but I hope is helpful). And if you like the approach and want to keep going in that way, I would look into Nearest Neighbor Classifiers.

Since the data looks rather comically consistent -- that's quite a pattern -- I would start by choosing a simple functional form, and a parabola sounds like a great idea. Simple b - ax^2 sort of shape. You can do this in R with

fit <- lm(y~poly(x,2,raw=TRUE))

where y are your viewers, and x is time, for EACH video. You'll get back the polynomial coefficients.

Then, now that you have something where hopefully you've removed some of the "noise" from the shapes, I would choose a simple distance metric (say Euclidean to start) and measure the distance between your video-to-predict (VtP), and the first k points (k minutes of viewers) of all of the videos you've seen already. (To do this, you'll need to generate k points from the curves you've fit, to compare to the k points you have from your VtP. You COULD just compare points-to-points directly, but I think there will be more over-fitting, so this might be an important regularization.)

THEN, you do one of a few things. Either, you just choose the curve that's closest, and assume things will go like that until the end. You're done! But what if the curve isn't a great fit? Well, then you could choose the 5 nearEST neighbors, and average their parameters (weighted by their closeness, maybe -- lots and lots of tweaks possible), then predict with THAT curve, using

weighted_mean(a_close) - weighted_mean(b_close)x^2

The biggest problem with this approach is that assuming a parabola is quite strict, but conceptually it's easier to think about mixing several polynomials, than it is to think about mixing several splines, and the mixing will be important, probably. You'll want to express each new curve as some combination of the old curves, and it's necessary to have a mix-able way of expressing them, to do that.

You could just try to average the five nearest-neighbor-curves directly, as well, but it will be harder to extrapolate to trajectories unlike those you've seen before -- with a parametric form, that would be easier.

one_observation
  • 1,500
  • 11
  • 15
  • thx - I have a dumb question - how do I express the equation for the parabola from the R lm model? It returns two coefficients ```Estimate Std. Error t value Pr(>|t|) (Intercept) -3.405221 22.279492 -0.153 0.879 poly(t, 2, raw = TRUE)1 53.592396 0.659911 81.212 <0.0000000000000002 *** poly(t, 2, raw = TRUE)2 -0.311924 0.004094 -76.186 <0.0000000000000002 ***``` (e.g., in this case for an R2 of 0.97) – metalaureate Nov 06 '17 at 04:22
  • And is that one curve fitted per video, or one curve fitted to the data of all videos? – metalaureate Nov 06 '17 at 04:27
  • 1
    1) coef(lm) => intercept, x's coef, x^2's coef. 2) One curve per video. It's really like pre-processing, cleaning them up a little before you try to determine which one you like best. – one_observation Nov 06 '17 at 04:36
  • I hope it does! Curious to hear how it turns out. – one_observation Nov 07 '17 at 01:30