0

I have 40 univariate Time series which I am clustering with tslearn.

To determine a reasonable amount of clusters, I use the silhouette coefficient. However, I noticed that it is extremely unrobust, at it delivers different maxima.

I use dynamic time warping as distance measure and perform an minmax transformation to preprocess the time series

I cannot share the data, but my the df looks like this: (just a small piece)

time       | value | label
2020-01-01    1.3    10000
2020-01-02    1.9    10000
2020-01-01    0.5    20000
2020-01-02    1.2    20000

my code:

# imports
from tslearn.clustering import TimeSeriesKMeans, silhouette_score
from sklearn.preprocessing import minmax_scale
import pandas as pd

# get list of time series, perform minmax-transformation
ts = []
for ts_label in df[self.ts_col].unique():
    ts.append(minmax_scale(df.loc[df[label] == ts_label, 'value']))
    
# loop through different configurations for # of clusters and store the respective values for silhouette:
sil_scores = []
for n in range(2, 10):
km = TimeSeriesKMeans(n_clusters=n, metric="dtw")
km.fit(ts)
sil_scores.append(silhouette_score(ts, km.predict(ts), metric="dtw"))

# prepare resulting df
result_df = pd.DataFrame(data={
                "no_clusters": range(min_n, max_n+1),
                "silhouette_score": sil_scores,
        })

however, if I repeat this process for multiple times, I get different results: The highest value for silhouette_score is either at 2, 3 or 5 clusters (I tried this 11 times and got four times 2 / five time 3 / two times 5)

Is there an error in my code / methodology or is this a common problem of silhouette score?

Thanks in advance

bk_
  • 199
  • 1
  • 10
  • opened an issue on github: https://github.com/tslearn-team/tslearn/issues/278 – bk_ Jul 24 '20 at 14:08
  • 1
    I'm not testing your code but giving a very general comment. _Any_ internal clustering criterion should better analyzed graphically for sharp "elbows" rather than looking only on it extremum value (read "Comparing different k: priority of sharpness over extremum" [here](https://stats.stackexchange.com/a/358937/3277). Also, regular ("robust") results can be ever expected only when (1) there _are_ relatively clear-cut clusters in the population, and (2) the (representative) sample is sufficiently large. There may be other nuances, too. – ttnphns Jul 25 '20 at 20:22
  • 1
    if the series is dynamical, you could use correlation dimension. another option is to look at minimum entropy in the histogram – user312088 Feb 21 '21 at 18:08
  • Different results could be due "random_state" paramter not set in. Try passing as like -- km = TimeSeriesKMeans(n_clusters=n, metric="dtw", random_state=0) – Dilip Upadhyay Jan 20 '22 at 11:15

0 Answers0