1

I have a collection of real-world samples that I am trying to model and then forecast.

There are 10 datasets – each has an increasing number of data points (from 1 to 10). The points are equi-distant and each has some percentage of the total allocation for that set. For example, the first set has 1 data point with 100% allocation; second has 2 points with 50% each, etc. The sets appear to have a roughly normal distribution – but with a small skew to the right of the curve.

As the number of data points increases with each set, the x-axis expands and the distribution is spread thinner over the set (ie. the mean decreases and the curve becomes more flat (lower kurtosis?)).

I am using MathNet Numerics for C#. I have tried importing one of the datasets into a normal distribution and having the library calculate the mean, SD, skew, kurtosis, etc.

Firstly, given those parameters for a normal distribution, is it possible to reproduce the values for a given number of data points? e.g. Can a normal distribution be divided equally on the x-axis and the percentage allocation for each point be calculated?

Secondly, given the sample datasets, is there any way to model or find a function for the change between the datasets so that I may forecast the distribution for any number of data points? (e.g. given the shape of the distribution for 5 and then 10 datapoints, what would the distribution be for 15 or 20?).

I am not a statistician, so please forgive me if I have asked something obvious or if anything doesn't make sense.

user1151406
  • 13
  • 3

1 Answers1

1

The straight answer to Q1 is "yes", it is definitely possible to cut up an underlying normally distributed continuous variable into an ordinal variable with 1 to 10 levels. You need something that can tell you the cumulative distribution function (often called CDF) of a normal distribution with a given mean and variance (you only need these two parameters to characterise a normal distribution). Then you need to calculate the differences between the values this returns for your various bin cutoffs (as its straight return will be the cumulative probability of a value at X or lower).

I'm sorry I don't use C# but in R this would be something like the below. This is for a 10 point example, if the normal distribution you think is your underlying latent variable has a mean of 5 and variance of 2; and my bins are minus infinity to 1.5, 1.5 to 2.5, 2.5 to 3.5, ... , 9.5 to infinity. You only need the mean and variance to characterise a normal distribution.

> options(digits=2)
> x <- pnorm(1:10+0.5, 5, 2)*100
> x[10] <- 100            # otherwise is just 9.5 to 10.5, not infinity
> x                       # ie cumulative prob (in %) to each bin
 [1]   4  11  23  40  60  77  89  96  99 100    
> c(x[1], diff(x))        # differences between the cumulative probs
 [1]  4.0  6.6 12.1 17.5 19.7 17.5 12.1  6.6  2.8  1.2

Subsequently, the straight answer to Q2 is also "yes" there are definitely such methods but they should be used with caution and it is probably a little difficult just here to summarise all the pros and cons of the different ways of doing this.

It's also worth knowing that there are other methods for analysing this sort of ordinal data.

Peter Ellis
  • 16,522
  • 1
  • 44
  • 82
  • Thank you for your answer Peter. I had seen some stuff about CDF but didn't get that you could basically use it like a histogram to carve up the distribution. I think that might actually be enough for my purposes - I'm thinking I could just choose one of the datasets as the model, take its mean and variance as you say and then just use CDF to split it into as many data points as I need. For interest's sake though - you mention that there are other methods for analysing this kind of data. Is there any chance you could just name a few possibilities so I can Google them? Thanks again! – user1151406 Jan 17 '12 at 22:52
  • Ordinal regression would be my starting point. – Peter Ellis Jan 17 '12 at 23:18