1

I posted this question on stackoverflow.com and have not received any answer. In case I get an answer from one of them, I will inform on the other.

I am using describe() to summarize data before and after normalizing data.

import pandas as pd
import urllib3
from sklearn import preprocessing

decathlon = pd.read_csv("https://raw.githubusercontent.com/leanhdung1994/Deep-Learning/main/decathlon.txt", sep='\t')
decathlon.describe()

nor_df = decathlon.copy()
nor_df.iloc[:, 0:10] = preprocessing.scale(decathlon.iloc[:, 0:10])
nor_df.describe()

At first, I have

enter image description here

Then I have

enter image description here

Could you please explain why the display is different? In the second one, the number is, for example, $4.100000e+01$.

I feel that this display makes it hard to check that the mean and variance of each column are 0 and 1 respectively.

Akira
  • 381
  • 1
  • 9
  • What difference in the display are you referring to? – jkpate Oct 27 '20 at 09:17
  • @jkpate In the second one, the number is, for example, $4.100000e+01$. – Akira Oct 27 '20 at 09:18
  • Also see [How to read scientific notation](https://stats.stackexchange.com/questions/319637/how-to-read-scientific-notation-output-numbers-that-include-e) ... and many more on site. – Glen_b Oct 27 '20 at 09:39

1 Answers1

1

The second display uses scientific notation. For example, the first count is reported as $4.1 * 10^1 = 4.1 * 10 = 41$. The original display did not use scientific notation because there are not many significant digits, while the second display uses scientific notation because, after standardization, there are many significant digits. See this question for approaches to control the use of scientific notation in Pandas displays.

Zero mean

The first mean is reported as $-1.516402 * 10^{-16}$, a tiny number that is very close to zero. It is not exactly zero because of floating point instability.

Unit variance

Similarly, the first standard deviation is $1.012423 * 10^0 = 1.012423 * 1 = 1.012423$, which is greater than one than you would expect. This is because Pandas describe() uses Pandas std(), which defaults to using Bessel's correction for sample variance. We can recover the the uncorrected variance by multiplying by $\frac{N-1}{N}$. For example, for the first standard deviation, the uncorrected variance is: \begin{align} \sigma & = \frac{40}{41} (1.012423 ** 2)\\ & = 1.0000003228575611 \end{align}

jkpate
  • 1,183
  • 5
  • 8