1

Which value for the within-cluster sum of squares points can be accepted regarding a data set of 1000 tuples, 21 attributes (but only 3 are used now)?

I have used Euclidean distance is used, and a standard argument like I 500 which will never be reached. The seed was = 10.

WEKA (Explorer) was used.

Currently, the best value was 290, the worst was 5600, but till now I only performed 7 experiments (5600 points were obtained with all attributes and a k of 2 - monkey test).

jonsca
  • 1,790
  • 3
  • 20
  • 30
mnemonic
  • 123
  • 5
  • I don't quite follow your question, @Monkey (although, I don't know clustering too well, so that may be part of the reason). Are you asking if there's some arbitrary value (like, say, .05) that if your data cross it, they are good? – gung - Reinstate Monica Oct 11 '12 at 02:37
  • What are those "7 experiments"? Did you run the k-means algorithm several times on the same dataset? Also, of possible interest: [How to tell if data is “clustered” enough for clustering algorithms to produce meaningful results?](http://stats.stackexchange.com/q/11691/930); [Elbow criteria to determine number of cluster](http://stats.stackexchange.com/q/11175/930). – chl Oct 11 '12 at 10:46
  • the best way to get a good count of clusters would be using EM, but i was used to work with kmeans in addition, but thanks. the question was about wcss and not about the count of clusters. the count of clusters (e.g. for an attribute such as credit amount) can be used with different k arguments, a k of 7 can provide a good result (aspect->overview) with kMeans, but EM uses only 2 clusters (good point concerning separation). if data is 'clustered' depends on the goal which should be achieved regarding a given data set. – mnemonic Oct 18 '12 at 16:32
  • chl: to answer briefly your questions - yes, i used it (kmeans of weka) on the same data set. firstly and secondly, with all 21 attributes - different k arguments 'of course' -> bad wcss value. afterwards weka/kmeans was applied with three selected attributes using different arguments for k (in the range 2-10). however, using rapidminer (another data mining software) provided another results with kmeans - better centroid tables. concerning your urls: tnx, but see previous comment. – mnemonic Oct 19 '12 at 02:58

1 Answers1

1

There is no rule of thumb, as the values are not comparable across data sets. Not even across different normalizations of the data set or across algorithms. You can mostly use them to compare different runs of the same algorithm, at most with slightly changed parameters (to some extend, not even across different k!)

Say, you scale every axis by 10. The data set is virtually unchanged, yet the within cluster sum of squares probably goes up by a factor of 100.

Or you leave away dimensions. Obviously the distances will become smaller, and thus the sum will go down. Try selecting a single attribute, and k=100 - what is your result? Can you get below 290 this way? This doesn't mean it is better - it just shows that the scores are not comparable across different settings such as dimensionality and normalization.

Has QUIT--Anony-Mousse
  • 39,639
  • 7
  • 61
  • 96
  • 40 clusters -> 145,6 wcss, but i 'dislike' a good wcss value with a subpar count of clusters. using em helps to compare the clustering results. – mnemonic Oct 18 '12 at 16:37