What are real life examples of "non-parametric statistical models"?

Question

I am reading the Wikipedia article on statistical models here, and I am somewhat perplexed as to the meaning of "non-parametric statistical models", specifically:

A statistical model is nonparametric if the parameter set $\Theta$ is infinite dimensional. A statistical model is semiparametric if it has both finite-dimensional and infinite-dimensional parameters. Formally, if $d$ is the dimension of $\Theta$ and $n$ is the number of samples, both semiparametric and nonparametric models have $d \rightarrow \infty$ as $n \rightarrow \infty$. If $d/n \rightarrow 0$ as $n \rightarrow \infty$, then the model is semiparametric; otherwise, the model is nonparametric.

I get that if the dimension, (I take that to literally mean, the number of parameters) of a model is finite, then this is a parametric model.

What does not make sense to me, is how we can have a statistical model that has an infinite number of parameters, such that we get to call it "non-parametric". Furthermore, even if that was the case, why the "non-", if in fact there are an infinite number of dimensions? Lastly, since I am coming at this from a machine-learning background, is there any difference between this "non-parametric statistical model" and say, "non-parametric machine learning models"? Finally, what might some concrete examples be of such "non-parametric infinite dimensional models" be?

Answer 1

As Johnnyboycurtis has answerd, non-parametric methods are those if it makes no assumption on the population distribution or sample size to generate a model.

A k-NN model is an example of a non-parametric model as it does not consider any assumptions to develop a model. A Naive Bayes or K-means is an example of parametric as it assumes a distribution for creating a model.

For instance, K-means assumes the following to develop a model All clusters are spherical (i.i.d. Gaussian). All axes have the same distribution and thus variance. All clusters are evenly sized.

As for k-NN, it uses the complete training set for prediction. It calculates the nearest neighbors from the test point for prediction. It assumes no distribution for creating a model.

For more info:

1. http://pages.cs.wisc.edu/~jerryzhu/cs731/stat.pdf
2. https://stats.stackexchange.com/a/133841/86202
3. https://stats.stackexchange.com/a/133694/86202

Answer 2

I'm currently taking a course on Machine learning, where we use the following definition of nonparametric models: "Nonparametric models grow in complexity with the size of the data".

Parametric model

To see what it mean let's have a look at linear regression, a parametric model: There we try to predict a function parametrized in $ w \in ℝ ^d $: $$ f(x) = w^Tx $$ The dimensionality of w is independent of the number of observations, or the size of your data.

Nonparametric models

Instead kernel regression tries to predict the following function: $$ f(x) = \sum_{i=1}^n \alpha_i k(x_i, x) $$ where we have $n$ data points, $\alpha_i$ are the weights and $k(x_i, x)$ is the kernel function. Here the number of parameters $\alpha_i$ is dependent on the number of data points $n$.

The same is true for the kernelized perceptron: $$ f(x) = sign( \sum_{i=1}^n \alpha_i y_i k(x_i,x))) $$

Let's come back to your definition and say d was the number of $\alpha_i$. If we let $ n \to \infty $ then $d \to \infty$. That's exactly what the wikipedia definition asks for.

I took the kernel regression function from my lecture slides and the kernelized perceptron function from wikipedia: https://en.wikipedia.org/wiki/Kernel_method

Answer 3

So, I think you're missing a few points. First, and most importantly,

A statistical method is called non-parametric if it makes no assumption on the population distribution or sample size.

Here is a simple (applied) tutorial on some nonparmetric models: http://www.r-tutor.com/elementary-statistics/non-parametric-methods

A researcher may decide to use a nonparemtric model vs a parametric model, say, nonparamtric regression vs linear regression, is because the data violates assumptions held by the parametric model. Since you're coming from a ML background, I'll just assume you never learned the typical linear regression model assumptions. Here is a reference: https://statistics.laerd.com/spss-tutorials/linear-regression-using-spss-statistics.php

Violating assumptions can skew your parameter estimates, and ultimately increase the risk of invalid conclusions. A nonparametric model is more robust to outliers, nonlinear relationships, and does not depend on many population distribution assumptions, hence, can provide more trust worthy results when trying to make inferences or predictions.

For a quick tutorial on nonparametric regression, I recommend these slides: http://socserv.socsci.mcmaster.ca/jfox/Courses/Oxford-2005/slides-handout.pdf