1

Suppose I have a sequence like "ADTGESW". Each character in this sequence can attain a number of possible values, let's say 10. I can then one-hot encode this sequence and obtain a matrix with shape 10 by the length of the sequence:

$$ \begin{matrix} position_0 & position_1 & \cdots & position_n \\ 0 & 1 & \cdots & 0 \\ 1 & 0 & \cdots & 0 \\ \vdots & \vdots & \vdots & \vdots \\ 0 & 0 & \cdots & 1 \end{matrix} $$

Ideally, this could be the input of a Convolutional Neural Network (CNN).

Suppose now I do not have enough data point to use a CNN, so that I have to rely on something simpler like a Random Forests (RF). Would it still be possible to use such an input (i.e., a matrix) with RF? One might say to just flatten the matrix, but then I would lose the locality properties that are instead preserved with CNNs. Moreover, I might have many similar matrices (for different properties of the sequence and thus with different shapes).

wrong_path
  • 607
  • 6
  • 20
  • You can definitely use a matrix as your features for Random Forest. The only thing is that features are typically represented by columns and observations by rows. It seems that each character in your sequence is a separate sample/observation and the value it takes is then encoded as 10 features, so you should rotate the matrix. – AlexK May 11 '19 at 08:08
  • @AlexK Actually, each sequence is sort of an observation by itself. And I have many of these sequences. – wrong_path May 11 '19 at 08:10
  • Then you'll want to check how the package you plan to use requires for input to be structured. Scikit-learn (Python) requires features to be a 2-dimensional array, so you would have to flatten your 3-dimensional array (with sequence as first dimension) to a 2-d array (i.e., convert this type of 10-by-n matrix to a 1-by-10*n vector). You can then concatenate additional features. – AlexK May 11 '19 at 08:21
  • @AlexK Yes, but as I wrote that would probably cause some problems considering the type of object (a sequence). So I was looking for something similar to what CNNs do but with less data! – wrong_path May 11 '19 at 08:22
  • Well, CNNs have more parameters to estimate than Random Forests for the same amount of training data. How to handle overfitting (not having enough samples) is a separate issue and it can be dealt with in multiple ways. You should rephrase the question in that case and provide specifics about the size of your data. – AlexK May 11 '19 at 08:31

1 Answers1

2

Convolutional neural networks are efficient for data such as images, because they seek for common patterns independently of their location on the picture. This cannot be achieved with random forests, they would rather mimic a dense neural network, if you want to make such comparisons. Moreover, when using dense neural network for images you would actually need more data then with convolutional network, since you would need more parameters. Random forests would work in here could work as a proxy for convolutional network only after extensive feature engineering.

Notice also that you won't be able to use random forest with this kind of data. Random forest is an algorithm implemented for two-dimensional data number of samples $\times$ number of features, while you mention three-dimensional data number of samples $\times$ sequence length $\times$ number of features, so still you would need to flatten your data.

Flattening the matrix would not lead to losing any positional information. It would only if you used something like convolutional neural network, that slide in two dimensions through the matrix, or when using feature engineering that do something similar. In other cases, it is just about how do you store your data, and it doesn't matter if your data is

$$ X = \left( \begin{array}{cc} a & b \\ c & d \end{array} \right)\qquad % Y = \left( \begin{array}{c} a \\ b \\ c \\ d \end{array} \right) $$

with $X_{1,2} = Y_2 = b$, as far as the elements are consistently on the same place.

As a side comment, it is not true that your data "Ideally, [...] could be the input of a Convolutional Neural Network (CNN)". For such data, you could use one-dimensional convolutions, but using two-dimensional convolutions for one-hot encoded categorical data is rather uncommon solution.

Tim
  • 108,699
  • 20
  • 212
  • 390
  • If a flatten my matrices, I’ll end up with something like 2000 rows (observations) and 10000 columns (features). I feel like this would cause some issues. – wrong_path May 11 '19 at 08:38
  • @wrong_path what kind of issues exactly? – Tim May 11 '19 at 08:39
  • More experienced researchers in the field of ML, have warned me about having a small number of observations and thousands of features. I’m not exactly sure why, though. – wrong_path May 11 '19 at 08:41
  • @wrong_path in both cases you end up with exactly the same number of features. – Tim May 11 '19 at 08:47