What are some uses of logistic regression at scale?

Question

Many libraries that scale linear and logistic regression assume a tall-skinny design matrix (many samples, few features), but I don't understand why you would need billions of samples if your data has 250 features.

In what scenarios would more data help? It seems like, instead of using more computational resources, you could simply sub-sample the data and achieve comparable accuracy in scenarios where the feature count is relatively small.

When feature count is high, say 100k, would having samples on the order of billions help? How do we decide how many samples are needed for a given modeling problem?

Perhaps more data helps with imbalanced data? e.g. when performing binary logistic regression for outlier detection, where the occurrence of positive samples is very small.

Any help here would be greatly appreciated.

Answer 1

Facebook's revenue in 2020 was 86 billion dollars, mostly by selling ads. So if your large scale logistic regression model can improve the click-through rate by 0.01%, you would make facebook 8.6 million dollars per year (86000000000*0.0001), which is enough money to justify spending a couple of millions on some grad students or engineers to work on that.

I don't know if there is a use case for fitting 250 PARAMETERS on billions of samples. But fitting tens of thousands of parameters is quite common. A paper might showcase a method on the 250 parameter toy problem, but my guess would be the real motivation is to fit bigger models. Also, these papers usually don't just fit your vanilla logistic regression, but some kind of regularized generalized linear model and a lot of things can be expressed as regularized logistic regression or GLM.

Even without being Facebook, you can find a problem that could be expressed as a model with billions of samples and hundreds of thousands of parameters. A lot of things can explode really quickly if you allow them to. For example, on a map, each point in a grid is a separate observation that, of course, needs to be modeled to take spatial relationships into account. Nevertheless, all of a sudden, you might have millions of "data points." You might have a time series, where of course, each measurement is a separate data point, and if you measure every millisecond, or if you measure for a long time, all of a sudden, you have millions or billions of data points. If you measure how something spatial changes across time, possibly with respect to some variables, e.g., fMRI signal in a brain, EEG signal on a scalp, smog in a city, global temperature, algae on a lake... you can easily get a problem of a scale you were talking about.

These problems might not always be looked at as large-scale logistic regression problems, but if there was a technology to efficiently fit those models, use cases would come by themselves.

tl;dr: time series, maps, and images. Could you just downsample it? Yes, but you wouldn't need to.

Answer 2

Your general premise is correct: you don’t need more data than you need. But also, as noticed in the other answer and comments, it may be more tricky than you’re assuming. First of all you need large enough, high-quality sample of data. Using more data than is needed for the desired precision would be wasteful. On another hand, logistic regression is a pretty fast and cheap model to train, at least as compared with many other machine learning models, so you wouldn’t usually care that much. You want the libraries to be scalable for cases where it’s needed, but this doesn’t mean that you always need to push as much data as you can to the algorithm.