Statistical Analysis Stack Exchange
Statistical Analysis Stack Exchange

Questions tagged [in-sample]

22 questions
13
votes
1 answer

Difference between "in-sample" and "pseudo out-of-sample" forecasts

Is there an explicit difference between in-sample forecasts and pseudo out-of-sample forecasts. Both is meant in the context of evaluating and comparing forecasting models.
altabq
  • 665
  • 3
  • 6
  • 16
13
votes
1 answer

What is the difference between in sample error and training error, and intuition of optimism?

In the book Elements of Statistical Learning in Chapter 7 (page 228), the training error is defined as: $$ \overline{err} = \frac{1}{N}\sum_{i=1}^{N}{L(y_i,\hat{f}(x_i))} $$ Whereas in-sample error is defined as $$ Err_{in} =…
SpeedBirdNine
  • 679
  • 7
  • 14
3
votes
0 answers

In practice, do we distinguish between "in-sample" and "training" error?

In Elements of Statistical Learning, it distinguishes between "in-sample" and "training" error (Frankly, I found the chapter on errors to be very confusing, especially with how they use the different error terminology). Also see the question asked…
24n8
  • 847
  • 3
  • 13
3
votes
1 answer

Out of Sample and In Sample testing

I am very confused in testing regressions and know that there are many explanations available online, but I am still not getting anything it in my mind. Suppose I have daily data for past 100 days, I run a simple linear regression estimate the…
Meesha
  • 133
  • 1
  • 1
  • 5
2
votes
1 answer

What is the posterior in-sample vs the posterior out-of-sample?

I'm watching this video on Bayesian modelling for the stock market by Thomas Wiecki, Thomas has a slide with two posterior distribution over the mean parameter in his stock return model. Around 18:26 we see this slide: The phrase “in-sample” and…
user27886
  • 585
  • 5
  • 14
2
votes
1 answer

in-sample data vs out-of-sample data

I know that a train-validation-test splits the data into: a training dataset - obviously my "in-sample" data a validation dataset a test data set - obviously my "out-of-sample" data My question is: Should I refer to the validation dataset as…
KittyKenty
  • 55
  • 6
2
votes
1 answer

In which scenarios are the in-sample error and training error NOT the same?

In Elements of Statistical Learning, Chapter 7 (pages 228-229), the authors define the optimism of the training error rate as: $$ op\equiv Err_{in}-\overline{err} $$ With the training error $\overline{err}$ defined as: $$ \overline{err} =…
Skander H.
  • 10,602
  • 2
  • 33
  • 81
2
votes
0 answers

Optimism bias - alternative references

In Hastie & al's book Elements of Statistical Learning, there are two subsections covering insample prediction errors and optimism bias (section 7, p.228-230). Hastie & al explain that defining the insample prediction error as (1), we can take the…
Xavier
  • 41
  • 4
2
votes
2 answers

what does it mean when out of sample AUC is greater than in sample AUC?

I am fitting a logistic regression model on a data set with about 200,000 observation and 100 features. According to SAS output, the model converged correctly with an in-sample AUC of 0.85. However, when I applied the model on a few out of sample…
user1946504
  • 1,247
  • 3
  • 14
  • 17
1
vote
1 answer

In-Sample and Out-of-sample forecasting accuracy

I am currently doing my college final project. I forecasted national soybeans yield and used MAPE to calculate the in-sample and out-of-sample forecasting accuracy. The MAPE results showed that the in-sample forecasting accuracy is higher than the…
adin
  • 13
  • 2
1
vote
1 answer

How does GARCH compute the realized daily volatility to be compared to the output of the model, to compute in-sample MSE?

How do GARCH and GJR-GARCH models (as implemented in rugarch or in EViews) calculate the in-sample MSE if they use the time series of daily returns as the input and don't use a time series of daily variances: How do these models compute the realized…
BillB
  • 89
  • 6
1
vote
0 answers

AIC based model selection, hyperparameter optimization and in-sample prediction

I'm using AIC to perform model selection along with hyperparameters optimization. The exact setup is the following: I have two input variables (A and B), and a single target variable. All variables are time series. Target variable is modeled with…
tmf
  • 11
  • 2
1
vote
1 answer

Which one should I use for rolling forecast, dynamic or static?

I'm doing a rolling forecast using a fitted arma-garch model, but I'm confused regarding the rolling method, my window length is 1209 obs, and I roll 100 times, and each time I reset my window to discard the first obs and add the next obs (thus from…
nahsel
  • 23
  • 6
1
vote
0 answers

Data Partition for In sample and out of sample forecasting in neural network

I got confused with how to do data partition that reflects in sample and out of sample forecast when I do time series forecasting in neural network. What I understand is we have to divide data into: - training data set --> to estimate the model -…
rizkarizka
  • 11
  • 2
1
vote
0 answers

Can you compare in-sample vs. out-of-sample using a MSPE?

Is it possible to compare in-sample and out-of-sample forecasts by calculating a MSPE for each? For example, my in sample period is say 12/1987-12/2015, and my out-of-sample forecasting period is 12/1993-12/2015 (I use an expanding window to…
Ene
  • 11
  • 2
1
2