I am trying to estimate the performance of a machine learning model on time series data. I saw the example of model evaluation using Monte Carlo Estimates from the book "Data Mining With R Learning With Case Studies", which said:
According to https://en.wikipedia.org/wiki/Monte_Carlo_method Monte Carlo methods vary, but tend to follow a particular pattern: 1.Define a domain of possible inputs. 2.Generate inputs randomly from a probability distribution over the domain. 3.Perform a deterministic computation on the inputs. 4.Aggregate the results.
My understanding is in this example randomly select points means that the probability distribution is the uniform distribution. And then the training and testing sets are decided based on the randomly selected points. Then evaluates the model using these training and testing sets. Finally the expectation of evaluation results represents the expectation performance of the model.
My question now is how to choose the R value that can be sufficiently for evaluating the model?
What I think the normal Monte Carlo methods are like Monte Carlo simulation or Monte Carlo integration that need to make use of many distributions and more complicated calculations. And here I think this example are more likely to apply the concept of Monte Carlo to the way of model evaluation.
