How Many Clusters? Picking K on the ML Map
Learn how SurgeFlow decides how many groups to sort stocks into, and why a stock's group can change from one day to the next.
Category: Machine Learning
In plain English
Imagine tipping a box of LEGO bricks on the floor and sorting them into piles. Should you make 6 piles or 10? Too few and every pile is a jumble; too many and each pile is nearly empty. A computer faces the same choice when it sorts stocks into groups by how they behave. This article explains how SurgeFlow picks the number of groups, and why a stock can sit in one group today and a different one tomorrow.
What it measures
A 'cluster' is just a group of stocks that look similar by the numbers (things like how fast the price has moved, how jumpy it is, and how big the company is). 'K' is how many clusters you make. To choose K, SurgeFlow scores each option with a 'silhouette score' — a number from -1 to +1 that asks: is each stock closer to its own group than to the nearest other group? Higher means tidier, better-separated groups. On its own, though, the silhouette can be fooled — so SurgeFlow adds a rule that no single group may hold more than about 40% of the stocks. - K = the number of groups; SurgeFlow tries a range (6 to 10) and keeps the tidiest - Silhouette near +1 = clean, well-separated groups; near 0 = blurry overlap - Tiny example: 100 stocks. Plain silhouette might dump 90 into one mega-blob and leave scraps — 'tidy' on paper but useless - The max-share cap forces a more useful split, e.g. several groups of varied sizes instead of one giant one
How SurgeFlow uses it
This is the engine behind the ML 'market map' tab. Each day SurgeFlow turns each stock into a row of numbers, then runs three different grouping methods (K-means, a Gaussian mixture, and hierarchical) and combines their votes into one agreed 'consensus' answer, so the result does not lean on any single method's quirks. The model is rebuilt from scratch every day, which keeps the map current but also means the groups are re-drawn each time. To tell a real change from a mere reshuffle, SurgeFlow matches today's groups to yesterday's by how much their membership overlaps (a technique called Hungarian matching) and gives each group a stable ID. - Three methods vote; the agreed answer is the 'consensus' cluster - Retrained daily — a fresh map every day, not a frozen one - Stable IDs link today's groups to yesterday's by membership overlap - So 'changed group' flags a genuine move in a stock's behaviour, not just a new label
How to read it on the site
Open the ML tab (the 'market map' at /ml) and pick a market. You will see the day's clusters as cards — click one to filter the table to just its members. A 'changed group' panel lists stocks that moved from one cluster to another since the last run, and a 'days in cluster' figure tells you how many daily runs a stock has stayed put. Treat all of this as a description of how stocks are grouping today, not a prediction or a buy/sell signal. - Cluster cards: click to see which stocks are in each group - 'Changed group': names that jumped clusters since the previous run - 'Days in cluster': consecutive daily runs in the same group (not calendar days) - Group names (e.g. 'high-vol', 'deep-value') are auto-generated labels, not verdicts
What it cannot tell you
Clustering describes structure — which stocks look alike right now — it does not forecast returns or tell you what to buy or sell. The number of groups is a judgement call, not a law of nature: a different K, or a slightly different set of stocks in the sample, can shift the picture. Because the model is rebuilt daily, the IDs are best-effort, not permanent — if many new stocks enter or leave the sample, groups can split, merge, or shuffle for reasons that have nothing to do with any one company. A 'changed group' flag is a prompt to look closer, never a conclusion on its own. - Not advice and not a forecast — it describes today, it does not predict tomorrow - Group count and shape depend on the chosen K and the current sample - IDs are stable best-effort, not permanent; a changing sample can reshuffle groups - Similar behaviour is not the same as a shared cause — correlation is not causation