Short text clustering
Hey folks, I’ve been using thesentence-transformerslibrary for trying to group together short texts.I’ve had reasonable success using theAgglomerativeClusteringlibrary from sklearn (using either euclidean distance +wardlinkage or precomputed cosine +averagelinkage) as it’s ability to set the distance thresholds + automatically find the right number of clusters (as opposed to Kmeans) is really nice.But while it seemingly provides great results during the first wave of clustering, it tends to struggle when finding decent groupings on outliers that slip through the net the first time (where there’s only 1-2 observations per group).I’ve tried some other clustering methods such as:KMedoidsHDBscan (not great)Kmeans / Agglomerative on predefined KBut none have been as effective as hierarchical on the initial embeddings. I’ve experimented with looping through and re-clustering just with slightly tighter distance thresholds on the outliers each time, but not really sure of a way to automatically set the distances without a large amount of trial and error.So I was wondering if anyone knew of any methods for:a) Grouping these more effectively during the first waveb) Better clustering together any of the remaining outliers after the first passAny help would be hugely appreciated - cheers!
2021-04-28T18:09:51Z
hey@scroobiustrip, have you tried first passing the embeddings through UMAP before applying a density based clustering algorithm? there’s a nice discussion on this approach in the UMAPdocswhich comes with the following warning:This is somewhat controversial, and should be attempted with care. For a good discussion of some of the issues involved in this, please see the various answersin this stackoverflow threadon clustering the results of t-SNE. Many of the points of concern raised there are salient for clustering the results of UMAP. The most notable is that UMAP, like t-SNE, does not completely preserve density. UMAP, like t-SNE, can also create false tears in clusters, resulting in a finer clustering than is necessarily present in the data.the example in the docs actually seems to exhibit some of the problems you found with e.g. HDBSCAN in your application
2021-04-29T15:41:31Z
Cheers@lewtunthat’s ace - I had attempted before, but didn’t really have much luck with finding the right parameters - I’m currently attempting with the following settings:umap_data = umap.UMAP(n_neighbors=5, n_components=1, spread=0.5, min_dist=0.0, metric='cosine').fit_transform(embeddings)

hdb = HDBSCAN(min_cluster_size=3,
              min_samples=5,
              metric='euclidean',                      
              cluster_selection_method='eom').fit(umap_data)But it’s leaving me with one very large cluster filled with outliers - with the rest grouped into fairly decent clusters in terms of quality - so just need to find a way of breaking this large blob down into more sensible sub-groupings I think. Thanks again for your help!
2021-04-30T19:23:15Z
perhaps you’ve already tried this, but does reducingn_neighbourshelp break apart the large cluster? it also seems that you’re projecting down to 1 dimension withn_components=1, so maybe you get better separate in a higher dimensional space (harder to visualise of course)another idea could be to combine two UMAP models (docs), one to target the large cluster, the other to tackle the remaining clusters.