unique.Make and the Weak Pointer Map: Canonicalizing Strings Without Leaking

TL;DR: unique.Make("hello") does not put your string in a map that lives forever. It hashes the value into a lock-free 16-way hash trie whose leaves hold a weak.Pointer to the canonical copy, and registers a cleanup callback. When the last Handle goes away the entry is deleted.

Interning strings is one of those things every language eventually grows. The idea is simple: if a program has ten thousand copies of the string "application/json", you want one copy in memory and ten thousand pointers to it. Comparison gets cheaper, memory gets smaller, and the GC has less work to do.

The problem is that a naive interner is a memory leak with extra steps. A global map[string]string never shrinks. Entries live forever because the map is a root. You can add finalizers, but finalizers are asynchronous, they resurrect the object, and they have never been a good fit for “drop this from the table”.

Go’s unique package (added in Go 1.23) takes a different route. It leans on two pieces of modern runtime machinery: weak pointers (weak.Pointer, Go 1.24) and cleanups (runtime.AddCleanup, Go 1.24). The result is an interner that is both fast and leak-free.

This is a walk through how it works, based on reading the Go 1.28 development tree.

The API is three methods

go
// src/unique/handle.go:15
// Handle is a globally unique identity for some value of type T.
//
// Two handles compare equal exactly if the two values used to create the handles
// would have also compared equal. The comparison of two handles is trivial and
// typically much more efficient than comparing the values used to create them.
type Handle[T comparable] struct {
	value *T
}

// Value returns a shallow copy of the T value that produced the Handle.
// Value is safe for concurrent use by multiple goroutines.
func (h Handle[T]) Value() T {
	return *h.value
}

That is the entire public surface of the unique package: Handle[T], Handle.Value(), and Make. The whole trick is in Make:

go
// src/unique/handle.go:30
// Make returns a globally unique handle for a value of type T. Handles
// are equal if and only if the values used to produce them are equal.
// Make is safe for concurrent use by multiple goroutines.
func Make[T comparable](value T) Handle[T] {
	// Find the map for type T.
	typ := abi.TypeFor[T]()
	if typ.Size() == 0 {
		return Handle[T]{(*T)(unsafe.Pointer(&zero))}
	}
	ma, ok := uniqueMaps.Load(typ)
	if !ok {
		m := &uniqueMap[T]{canonMap: newCanonMap[T](), cloneSeq: makeCloneSeq(typ)}
		ma, _ = uniqueMaps.LoadOrStore(typ, m)
	}
	m := ma.(*uniqueMap[T])

	// Find the value in the map.
	ptr := m.Load(value)
	if ptr == nil {
		// Insert a new value into the map.
		ptr = m.LoadOrStore(clone(value, &m.cloneSeq))
	}
	return Handle[T]{ptr}
}

Read that bottom half carefully. m.Load(value) looks the value up. If it is not there, we clone it and insert it. If it is there, we throw the clone away and return the pointer that was already in the table. That is the whole canonicalization contract: the second unique.Make("hello") returns a handle whose Value() points at the same bytes as the first one.

Note also typ.Size() == 0: the empty struct. Every empty struct is equal to every other empty struct, so there is exactly one canonical one, and it is a pointer to a package-level zero uintptr rather than a heap allocation.

Two levels of dispatch

uniqueMaps is an internal/sync.HashTrieMap keyed by *abi.Type:

go
// src/unique/handle.go:55
// uniqueMaps is an index of type-specific concurrent maps used for unique.Make.
//
// The two-level map might seem odd at first since the HashTrieMap could have "any"
// as its key type, but the issue is escape analysis. We do not want to force lookups
// to escape the argument, and using a type-specific map allows us to avoid that where
// possible (for example, for strings and plain-ol'-data structs). We also get the
// benefit of not cramming every different type into a single map, but that's certainly
// not enough to outweigh the cost of two map lookups. What is worth it though, is saving
// on those allocations.
var uniqueMaps isync.HashTrieMap[*abi.Type, any] // any is always a *uniqueMap[T].

The comment explains the odd-looking indirection: one map per type means the compiler can generate a type-specialised map implementation for each T, so hashing a string uses the runtime’s string hasher directly instead of hashing an any box. The cost is one extra lookup in uniqueMaps; the benefit is that the per-value lookup never has to box.

The canonical map is a hash trie, not a Go map

Here is where the design gets interesting. You might expect unique to use an ordinary map[T]*T with weak keys. It does not — it rolls its own trie:

go
// src/unique/canonmap.go:17
// canonMap is a map of T -> *T. The map controls the creation
// of a canonical *T, and elements of the map are automatically
// deleted when the canonical *T is no longer referenced.
type canonMap[T comparable] struct {
	root atomic.Pointer[indirect[T]]
	hash func(unsafe.Pointer, uintptr) uintptr
	seed uintptr
}

func newCanonMap[T comparable]() *canonMap[T] {
	cm := new(canonMap[T])
	cm.root.Store(newIndirectNode[T](nil))

	var m map[T]struct{}
	mapType := abi.TypeOf(m).MapType()
	cm.hash = mapType.Hasher
	cm.seed = uintptr(runtime_rand())
	return cm
}

Three things to notice.

First, the hasher is borrowed from the runtime’s map implementation for type T. mapType.Hasher is the exact function the runtime uses to hash map[T] keys, so unique gets the fast, assembly-optimised string hasher for free. The seed is random per map (runtime.rand, pulled in with a //go:linkname so the package does not have to depend on math/rand/v2), which is the same defence against hash-flooding that Go maps have.

Second, abi.NoEscape is used when hashing, so the key does not escape to the heap just because we looked at it.

Third, and most importantly, this is not a map[T]*T. It is a hand-rolled hash trie, because the entries need to be individually removable when the value they point to dies. A Go map cannot do that; you cannot delete an entry whose key you can no longer compare against.

The trie itself is a 16-way radix tree over the bits of the hash:

go
// src/unique/canonmap.go:275
const (
	// 16 children. This seems to be the sweet spot for
	// load performance: any smaller and we lose out on
	// 50% or more in CPU performance. Any larger and the
	// returns are minuscule (~1% improvement for 32 children).
	nChildrenLog2 = 4
	nChildren     = 1 << nChildrenLog2
	nChildrenMask = nChildren - 1
)

// indirect is an internal node in the hash-trie.
type indirect[T comparable] struct {
	node[T]
	dead     atomic.Bool
	parent   *indirect[T]
	mu       sync.Mutex // Protects mutation to children and any children that are entry nodes.
	children [nChildren]atomic.Pointer[node[T]]
}

Lookups are lock-free — just atomic loads down the trie:

go
// src/unique/canonmap.go:37
func (m *canonMap[T]) Load(key T) *T {
	hash := m.hash(abi.NoEscape(unsafe.Pointer(&key)), m.seed)

	i := m.root.Load()
	hashShift := 8 * goarch.PtrSize
	for hashShift != 0 {
		hashShift -= nChildrenLog2

		n := i.children[(hash>>hashShift)&nChildrenMask].Load()
		if n == nil {
			return nil
		}
		if n.isEntry {
			v, _ := n.entry().lookup(key)
			return v
		}
		i = n.indirect()
	}
	panic("unique.canonMap: ran out of hash bits while iterating")
}

Each level consumes four bits of the hash and picks one of sixteen children. A nil child means the key is definitely absent. A leaf node is an entry.

The leaf holds a weak pointer

Here is the heart of the design:

go
// src/unique/canonmap.go:307
// entry is a leaf node in the hash-trie.
type entry[T comparable] struct {
	node[T]
	overflow atomic.Pointer[entry[T]] // Overflow for hash collisions.
	key      weak.Pointer[T]
	hash     uintptr
}

func newEntryNode[T comparable](key T, hash uintptr) (*entry[T], *T, weak.Pointer[T]) {
	k := new(T)
	*k = key
	wp := weak.Make(k)
	return &entry[T]{
		node: node[T]{isEntry: true},
		key:  wp,
		hash: hash,
	}, k, wp
}

The entry stores the key weakly. It does not hold a strong reference to the canonical *T. That is the entire trick: the trie is not a GC root for the values it indexes, so the values can die, and when they do, the entries must be removed.

The overflow chain handles full hash collisions — two different values with the same 64-bit hash live in a linked list hanging off the same leaf.

Lookup walks that chain, and a dead entry (one whose weak pointer has been cleared) is simply skipped:

go
// src/unique/canonmap.go:326
// lookup finds the entry in the overflow chain that has the provided key.
//
// Returns the key's canonical pointer and the weak pointer for that canonical pointer.
func (e *entry[T]) lookup(key T) (*T, weak.Pointer[T]) {
	for e != nil {
		s := e.key.Value()
		if s != nil && *s == key {
			return s, e.key
		}
		e = e.overflow.Load()
	}
	return nil, weak.Pointer[T]{}
}

s := e.key.Value() returns nil if the canonical value has been collected. That is the “this entry is dead” signal, and it is checked on every single lookup.

Insert, prune, and register the cleanup

Insertion is where the lock comes in. The writer walks the trie without a lock, finds a candidate slot, takes the lock on that node, double-checks, and then either replaces the leaf or expands it into a subtree:

go
// src/unique/canonmap.go:121
	newEntry, canon, wp := newEntryNode(key, hash)
	// Prune dead pointers. This is to avoid O(n) lookups when we store the exact same
	// value in the set but the cleanup hasn't run yet because it got delayed for some
	// reason.
	oldEntry = oldEntry.prune()
	if oldEntry == nil {
		// Easy case: create a new entry and store it.
		slot.Store(&newEntry.node)
	} else {
		// We possibly need to expand the entry already there into one or more new nodes.
		//
		// Publish the node last, which will make both oldEntry and newEntry visible. We
		// don't want readers to be able to observe that oldEntry isn't in the tree.
		slot.Store(m.expand(oldEntry, newEntry, hash, hashShift, i))
	}
	runtime.AddCleanup(canon, func(_ struct{}) {
		m.cleanup(hash, wp)
	}, struct{}{})
	return canon
}

Two things stand out.

The prune. Before inserting, the code walks the overflow chain and drops every entry whose weak pointer is already dead. The comment explains why: without pruning, inserting a value that was recently interned and recently dropped would have to walk past a dead entry every time. Pruning keeps the chain short.

The cleanup. runtime.AddCleanup(canon, ...) registers a callback on the canonical *T. The runtime guarantees the callback runs in a separate goroutine some time after canon becomes unreachable. The closure captures the map, the hash, and the weak pointer — and crucially not canon itself, because a cleanup that referenced its own object would never run.

So the lifecycle is:

text
   unique.Make("x")            last Handle dropped
        |                              |
        v                              v
   canonical *T  <--- strong ref ---  Handle
        |                              |
        |  (map holds only a weak ref) GC frees it
        v                              |
   weak.Pointer[T].Value() == nil      |
        |                              |
        +--> runtime cleanup fires ----+
                     |
                     v
             m.cleanup(hash, wp)
                     |
                     v
             trie entry deleted

Deleting the entry

The cleanup function walks the trie by hash, finds the leaf, verifies the weak pointer matches, and removes the entry — pruning empty interior nodes on the way up:

go
// src/unique/canonmap.go:176
// cleanup deletes the entry corresponding to wp in the canon map, if it's
// still in the map. wp must have a Value method that returns nil by the
// time this function is called. hash must be the hash of the value that
// wp once pointed to (that is, the hash of *wp.Value()).
func (m *canonMap[T]) cleanup(hash uintptr, wp weak.Pointer[T]) {
	var i *indirect[T]
	var hashShift uint
	var slot *atomic.Pointer[node[T]]
	var n *node[T]
	for {
		// Find wp in the map by following hash.
		i = m.root.Load()
		hashShift = 8 * goarch.PtrSize
		haveEntry := false
		for hashShift != 0 {
			hashShift -= nChildrenLog2

			slot = &i.children[(hash>>hashShift)&nChildrenMask]
			n = slot.Load()
			if n == nil {
				// We found a nil slot, already deleted.
				return
			}
			if n.isEntry {
				if !n.entry().hasWeakPointer(wp) {
					// The weak pointer was already pruned.
					return
				}
				haveEntry = true
				break
			}
			i = n.indirect()
		}

The rest of the function takes the node’s lock, prunes the leaf, stores nil into the slot if the chain is now empty, and then walks up the tree deleting interior nodes that have become empty — locking each parent in turn, because unlinking a child requires the parent’s lock.

This is the part that makes the design safe under concurrency. Two goroutines can intern the same value at the same time; the second one finds the entry via lookup and returns the first one’s pointer. Two goroutines can drop handles at the same time; the cleanup runs once (the weak pointer match in hasWeakPointer guards against a double-delete), and if the entry was already pruned by somebody else, it returns harmlessly.

The clone trick

There is one more subtlety, and it is the one that surprises people. When unique interns a value, it does not keep the caller’s copy. It makes its own copy first:

go
// src/unique/clone.go:13
// clone makes a copy of value, and may update string values found in value
// with a cloned version of those strings. The purpose of explicitly cloning
// strings is to avoid accidentally giving a large string a long lifetime.
//
// Note that this will clone strings in structs and arrays found in value,
// and will clone value if it itself is a string. It will not, however, clone
// strings if value is of interface or slice type (that is, found via an
// indirection).
func clone[T comparable](value T, seq *cloneSeq) T {
	for _, offset := range seq.stringOffsets {
		ps := (*string)(unsafe.Pointer(uintptr(unsafe.Pointer(&value)) + offset))
		*ps = stringslite.Clone(*ps)
	}
	return abi.EscapeToResultNonString(value)
}

Why? Consider unique.Make(bigStruct). The struct contains a string field holding a 10 MB buffer. If the canonical copy pointed at the caller’s bytes, then interning the struct would pin 10 MB for as long as any handle to the struct existed — even if the original owner had long since dropped its reference. Cloning the string first means the canonical value owns its own bytes, and the original 10 MB can be freed independently.

cloneSeq is precomputed per type so that the clone is a fixed sequence of pointer writes rather than a reflective walk:

go
// src/unique/clone.go:29
// singleStringClone describes how to clone a single string.
var singleStringClone = cloneSeq{stringOffsets: []uintptr{0}}

// cloneSeq describes how to clone a value of a particular type.
type cloneSeq struct {
	stringOffsets []uintptr
}

// makeCloneSeq creates a cloneSeq for a type.
func makeCloneSeq(typ *abi.Type) cloneSeq {
	if typ == nil {
		return cloneSeq{}
	}
	if typ.Kind() == abi.String {
		return singleStringClone
	}
	var seq cloneSeq
	switch typ.Kind() {
	case abi.Struct:
		buildStructCloneSeq(typ, &seq, 0)
	case abi.Array:
		buildArrayCloneSeq(typ, &seq, 0)
	}
	return seq
}

And the crucial part, back in Make: the clone only happens on the miss path. If the value is already in the table, m.Load(value) returns the existing canonical pointer and clone is never called. So the second unique.Make("hello") allocates nothing and copies nothing — it hands back the same bytes.

What `Handle.Value()` guarantees

Handle[T] holds a plain *T. Two handles to the same value hold the same pointer. Value() dereferences it and returns a copy of the T.

For a string that means the returned string header points at the canonical bytes. Two handles’ Value() results compare equal, and their underlying byte slices are literally the same allocation. That is the memory saving.

The handle itself is a strong reference, and that is what keeps the canonical value alive. As long as you hold a Handle, the value cannot be collected. When you drop the last handle, the value becomes collectable, the cleanup fires, and the table forgets it.

Trade-offs

What you get:

Deduplication. One copy of the bytes per distinct value, no matter how many handles point at it.
Fast comparison. Handle comparison is a pointer compare.
No leak. The table shrinks when the values do.
Lock-free reads. The fast path is a handful of atomic loads and a hash.

What you pay:

A hash on every Make. You are paying a hash plus a trie walk to save memory you might not have been going to use twice.
A cleanup callback per interned value. Every canonical value gets a runtime.AddCleanup registration, which is a special record on the span plus a queue entry when it fires.
Cleanup is asynchronous. The entry is removed some time after the value dies, not at the instant it dies. Between those two moments, the table still holds a dead entry, which is why the prune-on-insert logic exists.
Cloning costs a copy. The first Make of a string copies its bytes. If you were going to use the string only once, that copy is pure overhead.
The value must be comparable. unique.Make is not a general-purpose memoiser.

Caveats

unique is not a string cache; it is a value canonicaliser. The point is that the same value produces the same pointer, not that lookups are fast. If you build the key of a map out of a struct and never compare the handles, you have paid for nothing.

The cleanup is not guaranteed to run promptly. It runs “some time after” the object is unreachable, in a separate goroutine, and cleanups are not guaranteed to run at all before program exit. If you drop the last handle and immediately intern the same value again, you may get a new canonical copy, because the old entry has not been cleaned up yet — or you may get the old one back. Both are legal; the value is still canonical at any given moment.

Zero-sized types get a shared sentinel. unique.Make(struct{}{}) returns a handle to a package-level zero uintptr, not to a heap object.

Handles are not comparable across types. Handle[string] and Handle[[]byte] are different types and do not interoperate.

The weak pointer semantics bite. weak.Pointer.Value() may return nil at any time after the target becomes unreachable, and the runtime may batch tiny pointer-free objects together, in which case a weak pointer may never become nil. unique handles this correctly, but code that reads weak.Pointer directly needs runtime.KeepAlive discipline.

Wrapping up

The one thing to remember: the canonical map’s leaves hold weak pointers, and the deletion of an entry is driven by a runtime.AddCleanup callback on the canonical value. The map is not a root; it is a participant in the same lifetime dance as everything else.

The part people get wrong: unique.Make clones your string the first time it interns it, and reuses the canonical bytes only on the second and subsequent calls. Interning a value you only use once is a pure loss — you pay a copy, a hash, and a trie insert for nothing.

What to go do about it: measure your hit rate before reaching for unique. If your program interns the same few hundred strings repeatedly (a very common shape for parsers, routers, and protocol decoders), the dedup is real and the weak-pointer machinery pays for itself. If it interns values that are mostly unique, use a plain map and let the GC do its job.