Statistical Analysis Stack Exchange
Statistical Analysis Stack Exchange

Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

516 questions
30
votes
3 answers

Prerequisites for AIC model comparison

What are exactly the prerequisites, that need to be fulfilled for AIC model comparison to work? I just came around this question when I did comparison like this: > uu0 = lm(log(usili) ~ rok) > uu1 = lm(usili ~ rok) > AIC(uu0) [1] 3192.14 >…
Tomas
  • 5,735
  • 11
  • 52
  • 93
19
votes
3 answers

Pitfalls of linear mixed models

What are some of the main pitfalls of using linear mixed-effects models? What are the most important things to test/watch out for in assessing the appropriateness of your model? When comparing models of the same dataset, what are the most important…
Alan H.
  • 4,899
  • 4
  • 20
  • 19
15
votes
3 answers

Similarity of two discrete fourier tranforms?

In climate modelling, you're looking for a models that can adequately portray the Earth's climate. This includes showing patterns that are semi-cyclical: things like the El Nino Southern Oscillation. But model verification occurs generally over…
naught101
  • 4,973
  • 1
  • 51
  • 85
14
votes
2 answers

What if there is no true data-generating process?

I've been engaging in a number of forecasting efforts recently, and have rediscovered a well-known truth: That combinations of different forecasts are generally better than the forecasts themselves. In particular, the unweighted mean of forecasts is…
andrewH
  • 2,587
  • 14
  • 27
14
votes
2 answers

Comparing two linear regression models

I would like to compare two linear regression models which represent degradation rates of a mRNA over time under two different conditions. The data for each model collected independently. Here is the dataset. Time (hours) log(Treatment A) …
Rooz
  • 141
  • 1
  • 1
  • 3
14
votes
4 answers

What is the relationship between ANOVA to compare means of several groups and ANOVA to compare nested models?

I've so far seen ANOVA used in two ways: First, in my introductory statistics text, ANOVA was introduced as a way to compare means of three or more groups, as an improvement over pairwise comparison, in order to determine if one of the means has a…
Austin
  • 683
  • 7
  • 19
13
votes
1 answer

Difference between "in-sample" and "pseudo out-of-sample" forecasts

Is there an explicit difference between in-sample forecasts and pseudo out-of-sample forecasts. Both is meant in the context of evaluating and comparing forecasting models.
altabq
  • 665
  • 3
  • 6
  • 16
13
votes
5 answers

When to use multiple models for prediction?

This is a fairly general question: I have typically found that using multiple different models outperforms one model when trying to predict a time series out of sample. Are there any good papers that demonstrate that the combination of models will…
Shane
  • 11,961
  • 17
  • 71
  • 89
13
votes
2 answers

Number of parameters in Markov model

I want to use BIC for HMM model selection: BIC = -2*logLike + num_of_params * log(num_of_data) So how do I count the number of parameters in the HMM model. Consider a simple 2-state HMM, where we have the following data: data = [1 2 1 1 2 2 2 1 2 3…
Sergey
  • 606
  • 6
  • 10
11
votes
1 answer

Comparison negative binomial model and quasi-Poisson

I have run negative binomial and quasi-Poisson models based on an hypothesis testing approach. My final models using both methods have different covariates and interactions. It seems that there are no patterns when I plot my residuals in both cases.…
Elena Spark
  • 375
  • 3
  • 12
11
votes
3 answers

Comparing regression models on count data

I recently fit 4 multiple regression models for the same predictor/response data. Two of the models I fit with Poisson regression. model.pois <- glm(Response ~ P1 + P2 +...+ P5, family=poisson(), ...) model.pois.inter <- glm(Response ~ (P1 + P2…
Daniel Standage
  • 1,109
  • 3
  • 13
  • 21
9
votes
1 answer

Evaluating Time Series Prediction Performance

I have a Dynamic Naive Bayes Model trained on a couple of temporal variables. The output of the model is the prediction of P(Event) @ t+1, estimated at each t. The plot of P(Event) versus time is as given in the figure below. In this figure, the…
Zhubarb
  • 7,753
  • 2
  • 28
  • 44
9
votes
0 answers

How to calculate percent partial deviance explained by each predictor variable in a GAM model?

I am trying to find a sensible way to calculate the deviance explained by each predictor variable in a GAM model and need some input on my calculations. Following Simon Wood's example on the thread…
Isabella Ghement
  • 18,164
  • 2
  • 22
  • 46
9
votes
3 answers

Is overfitted model with higher AUC on test sample better than not overfitted one

i am participating in a challange in which I have created a model that performs 70% AUC on train set and 70% AUC on hold-out test set. The other participant has created a model that performs 96% AUC on train set and 76% AUC on hold-out test set.…
MiksL
  • 177
  • 2
  • 8
9
votes
3 answers

Model comparison between an ARIMA model and a regression model

I'm really having trouble finding out how to compare ARIMA and regression models. I understand how to evaluate ARIMA models against each other, and different types of regression models (ie: regression vs dynamic regression with AR errors) against…
Brett
  • 91
  • 1
  • 1
  • 2
1
2 3
…
34 35