How simple should a Baseline model be?

Question

I was making baseline model and was wondering how much time I should spend on it. I've found a lot of article about the purpose of a baseline model, for instance;

A baseline takes only 10% of the time to develop, but will get us 90% of the way to achieve reasonably good results.

But how do I properly select one?

1. I'm I choosing among multiple algorithms/models?
2. I'm I suppose to tune my simple model or leave it with its default?

Answer 1

Mainly, how to do this is a question of experience. This will also tell you what kind of model is a good candidate for such a baseline. For instance, in time series forecasting, the simplest models, which are actually surprisingly hard to beat, are the historical average of a time series, and the last observation. Note that both models need precisely zero tuning!

An alternative approach would be to pick the simplest model that can at least output some "reasonable" figure. For time series forecasting, the simplest model would be one that always output a flat zero, but that is not "reasonable". So the next most simple would be one of the two above, and either one would be a good start.

Finally, you could also start by time-boxing your entire development effort, then allot the first 5-10% (or whatever is a reasonable number - I would use something closer to 5%, or even 2%, than to 10%) to building this simple benchmark. This allows for a little tuning, but keeps you from over-engineering something that is intended to only be a simple benchmark.

Answer 2

You should tune the model if best practice application of that technique demands it. The baseline needs to be competently implemented to be meaningful.

Hyper-parameter tuning can make a very significant difference, sometimes the difference between a state-of-the-art method and an un-competitive one, see e.g.

Anthony Bagnall, Gavin C Cawley "On the use of default parameter settings in the empirical evaluation of classification algorithms" arxiv

Just pick an algorithm you think would not be regarded as a straw man by an experienced practitioner (or reviewer 3 ;o) and apply it competently. Do bear in mind that the better your baseline system, the more eye-catching the results if your proposed method is significantly better.

Caveat lector, Gavin Cawley is my alter-ego.

Answer 3

There's multiple type of baseline models:

1. A first try that you can compare yourself to, to see whether you are even doing anything meaningful, at all. E.g. a super-simple logistic regression linear model, or even just something as simple as predicting the average sale of the last month as the sale for the next day. If you do not beat or even do worse than such a baseline, there is a problem (either you have messed something up, or perhaps there is no useful information for making a prediction). This is mostly a sanity check to protect yourself from wasting a lot of effort without achieving anything even marginally useful.
2. Very similar to 1, but using an option that many would regard as a basic default choice (e.g. LightGBM with parameters tuned using cross-validation for tabular data, modern convolutional neural network competently trained with suitable image augmentation for image classification etc.) without extra fancy add-ons (e.g. no model stacking of models from different model classes). An experienced person can put something like this together in as little as 1 day or perhaps up to a week (assuming it's a relatively standard use case, depending on the exact details and depending on how much the data/data loading still needs to be optimized etc.).
3. A comparison baseline that you wish to beat (or at least be close to) to demonstrate that a new method is useful. This would typically be the currently best option (e.g. the system that is currently used in an industry example) / state of the art (for academic publishing purposes).

Answer 4

I agree (+1) with the answers by Stephan Kolassa and Dikran Marsupial, but let me add my two cents.

What you need to consider as well is what is your target model. For example, for a classification task logistic regression may be a good benchmark, but if your actual model is logistic regression, you obviously would choose something simpler than that. Usual choice would be a simple model that is known to work for such problems, for example naive forecast for time-series, mentioned by Stephan Kolassa, or LSTM model for NLP task. If you were building a novel state-of-the-art model, you would benchmark against other state-of-the-art models. On another hand, it is always useful to have a trivial benchmark as a sanity check (predict mean, median, the most common value, last value, etc), because there is always risk that your simple benchmark is not good itself.