Statistical Analysis Stack Exchange
Statistical Analysis Stack Exchange

Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

Overview

Linear algebra is the field of mathematics concerned with the study of finite-dimensional vector spaces. Matrices and their manipulation, which form the algorithmic part of linear algebra, are particularly important in statistics.

References

The following are introductory references:

A thread focused on linear algebra references useful for applied statistics is the following:

The following threads from math.se also have lists of references:

589 questions
187
votes
10 answers

Why the sudden fascination with tensors?

I've noticed lately that a lot of people are developing tensor equivalents of many methods (tensor factorization, tensor kernels, tensors for topic modeling, etc) I'm wondering, why is the world suddenly fascinated with tensors? Are there recent…
Y. S.
  • 1,237
  • 3
  • 9
  • 14
80
votes
3 answers

What is the intuition behind SVD?

I have read about singular value decomposition (SVD). In almost all textbooks it is mentioned that it factorizes the matrix into three matrices with given specification. But what is the intuition behind splitting the matrix in such form? PCA and…
SHASHANK GUPTA
  • 1,139
  • 2
  • 10
  • 17
75
votes
6 answers

What is an intuitive explanation for how PCA turns from a geometric problem (with distances) to a linear algebra problem (with eigenvectors)?

I've read a lot about PCA, including various tutorials and questions (such as this one, this one, this one, and this one). The geometric problem that PCA is trying to optimize is clear to me: PCA tries to find the first principal component by…
stackoverflowuser2010
  • 3,190
  • 5
  • 27
  • 35
60
votes
5 answers

Is every covariance matrix positive definite?

I guess the answer should be yes, but I still feel something is not right. There should be some general results in the literature, could anyone help me?
Jingjings
  • 1,173
  • 1
  • 9
  • 13
57
votes
9 answers

Reference book for linear algebra applied to statistics?

I have been working in R for a bit and have been faced with things like PCA, SVD, QR decompositions and many such linear algebra results (when inspecting estimating weighted regressions and such) so I wanted to know if anyone has a recommendation on…
Palace Chan
  • 759
  • 3
  • 9
  • 17
45
votes
7 answers

Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

I am studying PCA from Andrew Ng's Coursera course and other materials. In the Stanford NLP course cs224n's first assignment, and in the lecture video from Andrew Ng, they do singular value decomposition instead of eigenvector decomposition of…
DongukJu
  • 593
  • 1
  • 5
  • 5
45
votes
4 answers

Why does inversion of a covariance matrix yield partial correlations between random variables?

I heard that partial correlations between random variables can be found by inverting the covariance matrix and taking appropriate cells from such resulting precision matrix (this fact is mentioned in http://en.wikipedia.org/wiki/Partial_correlation,…
michal
  • 1,138
  • 3
  • 11
  • 14
38
votes
3 answers

Distribution of scalar products of two random unit vectors in $D$ dimensions

If $\mathbf{x}$ and $\mathbf{y}$ are two independent random unit vectors in $\mathbb{R}^D$ (uniformly distributed on a unit sphere), what is the distribution of their scalar product (dot product) $\mathbf x \cdot \mathbf y$? I guess as $D$ grows the…
amoeba
  • 93,463
  • 28
  • 275
  • 317
34
votes
3 answers

Why is a sample covariance matrix singular when sample size is less than number of variables?

Let's say I have a $p$-dimensional multivariate Gaussian distribution. And I take $n$ observations (each of them a $p$-vector) from this distribution and calculate the sample covariance matrix $S$. In this paper, the authors state that the sample…
user34790
  • 6,049
  • 6
  • 42
  • 64
31
votes
7 answers

Why are symmetric positive definite (SPD) matrices so important?

I know the definition of symmetric positive definite (SPD) matrix, but want to understand more. Why are they so important, intuitively? Here is what I know. What else? For a given data, Co-variance matrix is SPD. Co-variance matrix is a important…
Haitao Du
  • 32,885
  • 17
  • 118
  • 213
26
votes
2 answers

Why is the Fisher Information matrix positive semidefinite?

Let $\theta \in R^{n}$. The Fisher Information Matrix is defined as: $$I(\theta)_{i,j} = -E\left[\frac{\partial^{2} \log(f(X|\theta))}{\partial \theta_{i} \partial \theta_{j}}\bigg|\theta\right]$$ How can I prove the Fisher Information Matrix is…
madprob
  • 363
  • 3
  • 6
26
votes
1 answer

Multivariate normal posterior

This is a very simple question but I can't find the derivation anywhere on the internet or in a book. I would like to see the derivation of how one Bayesian updates a multivariate normal distribution. For example: imagine that $$…
Alex
  • 743
  • 1
  • 9
  • 16
22
votes
1 answer

How to whiten the data using principal component analysis?

I want to transform my data $\mathbf X$ such that the variances will be one and the covariances will be zero (i.e I want to whiten the data). Furthermore the means should be zero. I know I will get there by doing Z-standardization and…
Angelorf
  • 1,441
  • 1
  • 13
  • 23
22
votes
2 answers

Why is the rank of covariance matrix at most $n-1$?

As stated in this question, the maximum rank of covariance matrix is $n-1$ where $n$ is sample size and so if the dimension of covariance matrix is equal to the sample size, it would be singular. I can't understand why we subtract $1$ from the…
user3070752
  • 365
  • 1
  • 2
  • 7
22
votes
1 answer

Updating SVD decomposition after adding one new row to the matrix

Suppose that I have a dense matrix $ \textbf{A}$ of $m \times n$ size, with SVD decomposition $$\mathbf{A}=\mathbf{USV}^\top.$$ In R I can calculate the SVD as follows: svd(A). If a new $(m+1)$-th row is added to $\mathbf A$, can one compute the new…
user1436187
  • 523
  • 2
  • 4
  • 10
1
2 3
…
39 40