Face recognition on a family archive sounds like a solved problem until you remember that faces in this archive change shape. A newborn and the ten-year-old they become don't just look different to a human — their face embeddings, the numeric vectors a recognition model produces, drift far enough apart that a single similarity threshold can't hold both ends of that journey at once. Run one global clustering pass over the whole archive with a threshold loose enough to catch a newborn and a ten-year-old as the same person, and it starts catching different people too. Tighten it enough to keep strangers apart, and every child gets split into several "people" as they age.
That's the core problem this post is about: not "how do you recognize a face," which is a mostly-solved problem with mature models, but "how do you track identity through the kind of change a face goes through over two decades of childhood," which is a different problem entirely and much less written about.
Windows first, then chain them
The approach that worked is to stop asking one clustering pass to span twenty years and instead ask a lot of small, easy clustering passes to each span six months, then chain the results together.
Step one: greedy clustering within half-year windows, at a strict similarity bar of 0.50 cosine. Within a six-month window, age drift is genuinely small — a child doesn't change enough face structure in six months to threaten a well-tuned clustering pass, so a strict bar can afford to be strict, and stay reliable.
Step two is the actual trick. Take the clusters from each half-year window and chain them across up to four adjacent windows, at a much looser bar of 0.42 cosine, using union-find to merge chains transitively. The logic: age drift that would break a direct match across a decade is small enough to bridge across a single half-year-to-half-year hop. So instead of ever needing one similarity check to span "newborn to age ten," the system only ever needs a chain of short hops, each easy on its own, that happen to add up to the long distance. Union-find is the right structure here because identity is transitive — if window A's cluster links to window B's, and B's links to C's, A and C should end up in the same group even though they were never compared directly.
Capping the chain at four adjacent windows (two years, given the half-year window size) is a deliberate limit, not an oversight. Let the loose 0.42 bar chain indefinitely and errors compound: a slightly-too-generous merge two hops in becomes the foundation for a slightly-too-generous merge four hops in, and eventually two different children end up sharing an identity because the chain drifted the whole way there one soft merge at a time. Capping the span keeps each chain's worst-case error bounded.
Junk clusters are worse than fewer clusters
Chaining generates candidate person-clusters, and not all of them are worth showing anyone. A purity gate runs after chaining and throws out clusters that don't look internally consistent enough to trust, and it's aggressive: on the founding archive, this cut the listed clusters from 545 down to 281. That's not a small trim, it's discarding more than half the raw output. The alternative — showing a person 545 candidate "people," a third of which are actually fragments or false merges — is worse than showing them fewer, cleaner ones. Nobody wants to clean up after a clustering algorithm; they want a shortlist that's mostly already right.
Corrections have to survive a re-cluster
None of this is a one-shot process. The clustering re-runs as an archive grows or a model improves, and every time it does, it would be unacceptable to lose the human corrections someone already made — renaming a cluster, merging two that were actually the same person, splitting one that wasn't. So every correction is written into a durable override table keyed by name, separate from the clustering output itself, and re-applied after every re-cluster. The clustering algorithm proposes; the override table has the final word. This is the same principle as version control for automatically generated code: never let the regenerable output silently clobber the part a person edited by hand.
The honest limit: siblings
The most useful thing I can say about this system is where it doesn't try to be clever. Siblings, in this archive, measure around 0.33 cosine similarity to each other — closer than two random strangers, but meaningfully below either the 0.50 or 0.42 bars the clustering actually uses. That's close enough that a more aggressive system could talk itself into merging them, and far enough that it would sometimes be wrong. Rather than gamble on that edge, the system treats near-miss similarity as a signal to ask, not an instruction to act: cases like this surface as a suggested merge for a human to confirm, never an automatic one. A wrong auto-merge silently corrupts two people's timelines at once; a suggestion that gets declined costs nothing.