Face recognition on a family archive sounds like a solved problem until you remember that faces in this archive change shape. A newborn and the ten-year-old they become don't just look different to a human, their face embeddings, the numeric vectors a recognition model produces, drift far enough apart that a single similarity threshold can't hold both ends of that journey. Run one global clustering pass over the whole archive with a threshold loose enough to catch a newborn and a ten-year-old as the same person, and it starts catching different people too. Tighten it enough to keep strangers apart, and every child gets split into several "people" as they age.
This post is about that problem. Recognizing a face is largely solved by mature models. Tracking identity through the change a face goes through over two decades of childhood is a different problem, and much less written about.
Windows first, then chain them
The approach that worked is to stop asking one clustering pass to span twenty years and instead ask a lot of small, easy clustering passes to each span six months, then chain the results together.
Step one: greedy clustering within half-year windows, at a strict similarity bar of 0.50 cosine. Within a six-month window, age drift is small. A child doesn't change enough in six months to threaten a well-tuned clustering pass, so a strict bar stays reliable.
Step two is the real trick. Take the clusters from each half-year window and chain them across up to four adjacent windows, at a much looser bar of 0.42 cosine, using union-find to merge chains transitively. The logic: age drift that would break a direct match across a decade is small enough to bridge in a single half-year hop. The system never needs one similarity check to span "newborn to age ten." It needs a chain of short hops, each easy on its own, that add up to the long distance. Union-find fits because identity is transitive: if window A's cluster links to B's, and B's links to C's, A and C end up in the same group even though they were never compared directly.
Capping the chain at four adjacent windows (two years, given the half-year window size) is deliberate. If the loose 0.42 bar chained indefinitely, errors would compound: a slightly generous merge two hops in becomes the foundation for another four hops in, and eventually two different children share an identity. Capping the span bounds each chain's worst-case error.
Junk clusters are worse than fewer clusters
Chaining generates candidate person-clusters, and not all of them are worth showing anyone. A purity gate runs after chaining and throws out clusters that don't look internally consistent enough to trust, and it's aggressive: on the founding archive, this cut the listed clusters from 545 down to 281, discarding more than half the raw output. Showing a person 545 candidate "people," a third of them fragments or false merges, is worse than showing fewer, cleaner ones. Nobody wants to clean up after a clustering algorithm; they want a shortlist that is mostly right.
Corrections have to survive a re-cluster
The clustering re-runs as an archive grows or a model improves, and it must not lose the corrections someone already made: renaming a cluster, merging two that were the same person, splitting one that wasn't. So every correction is written into a durable override table keyed by name, separate from the clustering output itself, and re-applied after every re-cluster. The clustering algorithm proposes; the override table has the final word. The principle is the same as for generated code: never let regenerable output overwrite what a person edited by hand.
The limit: siblings
Here is where the system doesn't try to be clever. Siblings in this archive measure around 0.33 cosine similarity to each other, closer than two random strangers, but well below the 0.50 and 0.42 bars the clustering uses. That is close enough that a more aggressive system could merge them, and far enough that it would sometimes be wrong. So near-miss similarity is treated as a reason to ask: these cases surface as a suggested merge for a human to confirm, never an automatic one. A wrong auto-merge silently corrupts two people's timelines at once; a suggestion that gets declined costs nothing.