3

I am writing some MapReduce code to calculate ordinary least squares from a sample of data. I'd like to include standard error, but I am running into a problem in calculating the variance of the noise. According to my reference, the variance is calculated by:

variance calculation

However, I don't compute $\hat\beta$ until the end. Thus, this estimate requires the entire sample set, which would defeat the point of MapReduce. If I wanted the standard error of the mean of a sample, for example, I could simply use an online variance calculation and aggregate the value until the end. Are there any alternatives for the OLS case where I could somehow aggregate a value throughout the job and use it at the end?

Nate
  • 131
  • 3
  • I think [online least squares regression](http://stats.stackexchange.com/questions/6920) could be adapted. Would this be the sort of solution you are looking for? Alternatively, solutions for relatively large $p$ might be generated through sequences of independent ordinary LS problems via ["matchers"](http://stats.stackexchange.com/a/46508/919). Because MapReduce is so general and you provide no information about typical values of $n$ or $p$, it's hard to know which (if either) of these approaches to investigate further. Maybe you could tell us more about your specific problem? – whuber Nov 10 '14 at 21:03
  • 1
    I will look into that. In general I expect the use case for this to be "tall and skinny", so n will be large and p will be small. This is designed for business intelligence type users who have lots of data and several (but not a ton of) columns. – Nate Nov 10 '14 at 21:10

0 Answers0