1

I have what seem like a silly question about dimensionality reduction.

I am just learning about this now on my own, and most of the information I have found is centered around reducing dimension by figuring out whether or not linear dependence is present. Upon learning about dimensionality reduction, what I sort of assumed was that one method would be to run a regression and figure out whether or not your variables were statistically significant or not. If they were not, then they could be eliminated, and this as well would reduce dimension. Is this also a correct method? Am I thinking about this the right way?

Please keep in mind that I am an undergraduate and have only taken introductory courses on regression, linear algebra, statistics, etc but I am interested in these topics and just trying to develop some intuition for different methods and when they are appropriate.

agra94
  • 287
  • 3
  • 11
  • It's not a silly question, but unfortunately it's one of those that (a) has a very large set of possible answers which (b) tend to be subtle and (c) have been dealt with extensively in other threads on this site. Check out some of the threads related to [variable selection in regression](https://stats.stackexchange.com/search?q=variable+selection+regression+significant). – whuber Jan 11 '18 at 23:49
  • There's nothing wrong with thinking of it as a regression problem. I can see how one would automatically relate dimension reduction to selecting the order of say polynomials. I recommend going through chapter 11 of "An Introduction to Multivariate Statistical Analysis," by T. W. Anderson. If you're not that into rigorous math, maybe have a look at Bishop's Pattern Recognition and Machine Learning. – idnavid Jan 12 '18 at 02:32

1 Answers1

2

In your post, what you are describing is feature selection. You can find many posts on methods for feature selection. Here are two popular posts

What most data scientist/statisticians see as dimension reduction is methods of actually condensing the information in your $m$ variables down to $p$ variables where $p < m$. This can be viewed as a method for feature selection as well. Here is a post that briefly talks about that

Two algorithms/methods that are commonly used for dimension reduction are principal component analysis (PCA) and Gaussian Random Projection.

I assume your goal is to select features for a regression model. Depending on your goal/research question, dimension reduction by PCA, or otherwise, may or may not be necessary. Often, interpretability is lost when using dimension reduction, so sticking to other methods of feature selection may behoove you if you'd like to keep interpretation in your statistical results.

Jon
  • 2,180
  • 1
  • 11
  • 28