I have glossed over this phrase many times without really understanding what it means. According to Wikipedia - Gaussian process
Gaussian processes can be seen as an infinite-dimensional generalization of multivariate normal distributions
What makes it infinite-dimensional? Just because the kernel function handles an arbitrary number of input dimensions?
While a sample from a multivariate Gaussian distribution produces a vector with a discrete number of elements, a sample from a Gaussian Process is a continuous function, which is "infinite-dimensional" in the sense that it is "indexed" by a continuously varying coordinate.
I'm not an expert in GPs, but I've found this page helpful.
Suppose we have $X \sim \mathcal N_n(\mu, \Sigma)$. We can think of $X$ as giving us a random function from $\{1, \dots, n\}$ to $\mathbb R$, which we evaluate by indexing so e.g. $X(1) = X_1$. The space of random functions with this domain is $n$-dimensional since it is spanned by the functions $\{e_1, \dots, e_n\}$ where $e_i(t) = \mathbf 1_{t=i}$ are just the standard basis vectors thought of as functions.
The stochastic process view of this is that we have an index set $T = \{1,\dots, n\}$ and then we have random variables $X_t : \Omega\to\mathbb R$ for each $t \in T$. A single realization of this process yields a sequence $(x_1, \dots, x_n)$ which can be thought of as a particular random function. More formally, if $(\Omega, \mathscr F, P)$ is our probability space, then a single realization of the stochastic process is the function from $T$ to $\mathbb R$ given by $t \mapsto X_t(\omega)$ where $\omega\in\Omega$ is the sample outcome.
If we want random functions with an infinite support (so what we more typically think of as functions, like $f : \mathbb R\to\mathbb R$ with $f(x)=x^2$) we can get those by using stochastic processes with larger index sets like $T=\mathbb N$ or $T = [0,\infty)$. A single realization of one of these processes gives us a function from $T$ to $\mathbb R$, but now these functions live in an infinite dimension space (typically). In other words, the space of functions that can be realized by this process is an infinite dimension function space, as opposed to $\mathcal N_n(\mu,\Sigma)$ where the space of realizable functions is finite dimensional.
If we further make the requirement that the outputs of our random functions have a multivariate Gaussian distribution for every finite collection of index points, then it turns out that this usefully generalizes the idea of a Gaussian distribution over a finite dimension space to an infinite dimension one.
In summary: a multivariate Gaussian gives us random functions in finite dimension spaces, a Gaussian Process can give us random functions from infinite dimension spaces.
Infinite dimensional Gaussian processes have sample functions which span an infinite dimensional space (subspace of Hilbert space of mean square integrable functions). Equivalently, the kernel expansion requires an infinite number of terms (Mercer's theorem). It is possible to have Gaussian random processes with countable or even uncountable index sets whose sample functions span a finite dimensional space, equivalently a kernel expansion with a finite number of terms, and hence are not infinite dimensional. So the previous answers are not correct
