One Allocation Per Object: The Reverse Marshal Inside the Protobuf Envelope

TL;DR: The Kubernetes protobuf wire format is a 4-byte magic prefix 0x6b 0x38 0x73 0x00 (“k8s”) followed by a runtime.Unknown message that carries the TypeMeta and the raw object bytes. Naively that is three allocations: marshal the object to []byte, build the Unknown around it, copy everything into a final buffer. Kubernetes does none of that. Unknown.NestedMarshalTo computes the exact final size up front and then marshals backwards from the end of the buffer, including the inner object, so the object’s bytes are never copied. On top of that, runtime.Allocator pools the buffer across an entire LIST or WATCH, so the same allocation serves every object in the response.

Every byte the API server writes to a client passes through a serializer, and the default choice is protobuf, not JSON:

// pkg/controlplane/apiserver/options/options.go:137
	// Overwrite the default for storage data format.
	s.Etcd.DefaultStorageMediaType = "application/vnd.kubernetes.protobuf"

That default applies to the storage format, but the same protobuf encoder is what a client gets when it sends Accept: application/vnd.kubernetes.protobuf — which every controller, every kubectl, and every aggregated API server does. So this is the hot path for every object the API server reads or writes. It is worth understanding exactly what happens to the bytes.

The envelope

The wire format is not “a protobuf-encoded Pod”. It is a protobuf-encoded runtime.Unknown, which is a generic envelope with four fields — the TypeMeta (typeMetaTag = 0xa), the raw object bytes (rawTag = 0x12), a content encoding (contentEncodingTag = 0x1a) and a content type (contentTypeTag = 0x22). The whole thing is preceded by a magic prefix:

// staging/src/k8s.io/apimachinery/pkg/runtime/serializer/protobuf/protobuf.go:34
var (
	// protoEncodingPrefix serves as a magic number for an encoded protobuf message on this serializer. All
	// proto messages serialized by this schema will be preceded by the bytes 0x6b 0x38 0x73, with the fourth
	// byte being reserved for the encoding style. The only encoding style defined is 0x00, which means that
	// the rest of the byte stream is a message of type k8s.io.kubernetes.pkg.runtime.Unknown (proto2).
	//
	// See k8s.io/apimachinery/pkg/runtime/generated.proto for details of the runtime.Unknown message.
	//
	// This encoding scheme is experimental, and is subject to change at any time.
	protoEncodingPrefix = []byte{0x6b, 0x38, 0x73, 0x00}
)

0x6b 0x38 0x73 is ASCII “k8s”, and the fourth byte is reserved for the encoding style — 0x00 being the only one defined. The prefix exists so the serializer can recognise its own output without parsing it: RecognizesData is a bytes.HasPrefix check, and the API server uses it to sniff which serializer a request body or a stored object is encoded with.

The envelope exists because the storage layer is type-agnostic. etcd does not know what a Pod is; it stores bytes plus a revision. The Unknown wrapper carries the GroupVersionKind alongside the payload so the API server can decode the object back into the right type later.

The naive implementation

The obvious way to build this is: marshal the object into a fresh []byte (allocation 1), set unk.Raw to it, marshal the Unknown into another fresh []byte (allocation 2), which copies the raw bytes in, then prepend the 4-byte prefix — either by allocating a third buffer and copying, or by a copy() into a larger one (allocation 3). Three allocations and two copies of the entire object payload, on every object, on every request. For a 100 MB LIST response that is 200 MB of needless copying.

The unbufferedMarshaller path in the serializer is exactly this, and it still exists for types that have not been regenerated with the efficient interface — staging/src/k8s.io/apimachinery/pkg/runtime/serializer/protobuf/protobuf.go:255 calls t.Marshal() into a fresh slice, assigns it to unk.Raw, and then marshals the Unknown into a second buffer sized by unk.Size(), which itself has to walk the raw bytes to measure them. The comment in the source says it plainly: “this path performs extra allocations”.

Computing the size up front

The fix is to know the final size before allocating, and to write everything in one pass. The size of the envelope is predictable:

// staging/src/k8s.io/apimachinery/pkg/runtime/serializer/protobuf/protobuf.go:331
// estimateUnknownSize returns the expected bytes consumed by a given runtime.Unknown
// object with a nil RawJSON struct and the expected size of the provided buffer. The
// returned size will not be correct if RawJSOn is set on unk.
func estimateUnknownSize(unk *runtime.Unknown, byteSize uint64) uint64 {
	size := uint64(unk.Size())
	// protobuf uses 1 byte for the tag, a varint for the length of the array (at most 8 bytes - uint64 - here),
	// and the size of the array.
	size += 1 + 8 + byteSize
	return size
}

unk.Size() gives the envelope’s own fields (TypeMeta, and the empty content-encoding/content-type strings, each of which still costs a tag byte and a varint). Then the raw payload costs: 1 byte for the rawTag, a varint for its length (bounded at 8 bytes for a uint64 — hence the +8 upper bound), and the payload itself.

The estimate is deliberately conservative: it over-allocates by up to 7 bytes for the length varint. Over-allocating by a handful of bytes is much cheaper than being wrong, and it lets the encoder skip the sovGenerated size computation for the length prefix.

Now the fast path:

// staging/src/k8s.io/apimachinery/pkg/runtime/serializer/protobuf/protobuf.go:237
	switch t := obj.(type) {
	case bufferedMarshaller:
		// this path performs a single allocation during write only when the Allocator wasn't provided
		// it also requires the caller to implement the more efficient Size and MarshalToSizedBuffer methods
		encodedSize := uint64(t.Size())
		estimatedSize := prefixSize + estimateUnknownSize(&unk, encodedSize)
		data := memAlloc.Allocate(estimatedSize)

		i, err := unk.NestedMarshalTo(data[prefixSize:], t, encodedSize)
		if err != nil {
			return err
		}

		copy(data, s.prefix)

		_, err = w.Write(data[:prefixSize+uint64(i)])
		return err

One Allocate, one NestedMarshalTo, one copy of the 4-byte prefix, one Write. The object payload is never copied: it is marshalled directly into the final buffer.

Marshalling backwards

Here is the heart of it. NestedMarshalTo computes the total message size first, then writes the fields from the end of the buffer backwards:

// staging/src/k8s.io/apimachinery/pkg/runtime/types_proto.go:160
// NestedMarshalTo allows a caller to avoid extra allocations during serialization of an Unknown
// that will contain an object that implements ProtobufMarshaller or ProtobufReverseMarshaller.
func (m *Unknown) NestedMarshalTo(data []byte, b ProtobufMarshaller, size uint64) (int, error) {
	// Calculate the full size of the message.
	msgSize := m.Size()
	if b != nil {
		msgSize += int(size) + sovGenerated(size) + 1
	}

	// Reverse marshal the fields of m.
	i := msgSize
	i -= len(m.ContentType)
	copy(data[i:], m.ContentType)
	i = encodeVarintGenerated(data, i, uint64(len(m.ContentType)))
	i--
	data[i] = contentTypeTag

The index i starts at msgSize — the end of the buffer — and marches downwards. Each field is written at the current position and then i is decremented past it. Because the content-type and content-encoding strings are the last fields in the protobuf encoding order, they are written first.

Then comes the important part — the inner object:

// staging/src/k8s.io/apimachinery/pkg/runtime/types_proto.go:181
	if b != nil {
		if r, ok := b.(ProtobufReverseMarshaller); ok {
			n1, err := r.MarshalToSizedBuffer(data[:i])
			if err != nil {
				return 0, err
			}
			i -= int(size)
			if uint64(n1) != size {
				// programmer error: the Size() method for protobuf does not match the results of LashramOt, which means the proto
				// struct returned would be wrong.
				return 0, fmt.Errorf("the Size() value of %T was %d, but NestedMarshalTo wrote %d bytes to data", b, size, n1)
			}
		} else {
			i -= int(size)
			n1, err := b.MarshalTo(data[i:])
			if err != nil {
				return 0, err
			}
			if uint64(n1) != size {
				// programmer error: the Size() method for protobuf does not match the results of MarshalTo, which means the proto
				// struct returned would be wrong.
				return 0, fmt.Errorf("the Size() value of %T was %d, but NestedMarshalTo wrote %d bytes to data", b, size, n1)
			}
		}
		i = encodeVarintGenerated(data, i, size)
		i--
		data[i] = rawTag
	}

r.MarshalToSizedBuffer(data[:i]) is handed the buffer at the current write position. The generated code for a Kubernetes type writes its fields backwards from the end of that slice — so the object lands in exactly the right place in the final buffer, in one pass, with no intermediate []byte and no copy.

The two branches correspond to the two marshaller interfaces, both of which require the caller to have already computed Size():

// staging/src/k8s.io/apimachinery/pkg/runtime/types_proto.go:25
type ProtobufMarshaller interface {
	// Size returns the number of bytes a call to MarshalTo would consume.
	Size() int
	// MarshalTo marshals to the start of the data buffer, which must be at least as big as Size(),
	// and returns the number of bytes written, which must be identical to the return value of Size().
	MarshalTo(data []byte) (int, error)
}

// ProtobufReverseMarshaller can precompute size, and marshals to the end of the provided data buffer.
type ProtobufReverseMarshaller interface {
	// Size returns the number of bytes a call to MarshalToSizedBuffer would consume.
	Size() int
	// MarshalToSizedBuffer marshals to the end of the data buffer, which must be at least as big as Size(),
	// and returns the number of bytes written, which must be identical to the return value of Size().
	MarshalToSizedBuffer(data []byte) (int, error)
}

That is the contract: the size is known, the buffer is pre-sized, and the marshal is a pure write into known space. The if uint64(n1) != size check is a guard against a type whose Size() disagrees with its marshal — a programmer error, not a runtime condition, which is why it returns an error rather than silently producing a corrupt message.

Finally the TypeMeta goes in at the front:

// staging/src/k8s.io/apimachinery/pkg/runtime/types_proto.go:209
	n2, err := m.TypeMeta.MarshalToSizedBuffer(data[:i])
	if err != nil {
		return 0, err
	}
	i -= n2
	i = encodeVarintGenerated(data, i, uint64(n2))
	i--
	data[i] = typeMetaTag
	return msgSize - i, nil

return msgSize - i — the number of bytes written is the distance from the final write position back to the start of the message. Because the write started at msgSize and ended at i, the message occupies data[0:msgSize] and the return value is msgSize in the happy path.

The generated code. The reverse-marshalling is not hand-written per type; it is generated. Here is the real Pod:

// staging/src/k8s.io/api/core/v1/generated.pb.go:8784
func (m *Pod) MarshalToSizedBuffer(dAtA []byte) (int, error) {
	i := len(dAtA)
	_ = i
	var l int
	_ = l
	{
		size, err := m.Status.MarshalToSizedBuffer(dAtA[:i])
		if err != nil {
			return 0, err
		}
		i -= size
		i = encodeVarintGenerated(dAtA, i, uint64(size))
	}
	i--
	dAtA[i] = 0x1a
	{
		size, err := m.Spec.MarshalToSizedBuffer(dAtA[:i])
		if err != nil {
			return 0, err
		}
		i -= size
		i = encodeVarintGenerated(dAtA, i, uint64(size))
	}
	i--
	dAtA[i] = 0x12

Every field writes itself at i and then steps i back past the tag and the length varint. The recursion is what makes the whole object land in one contiguous region: Pod.Spec.MarshalToSizedBuffer(dAtA[:i]) writes the spec just below the status, and so on down the tree.

The varint encoder has the same backwards property: encodeVarintGenerated (staging/src/k8s.io/apimachinery/pkg/runtime/generated.pb.go:150) moves the offset back by the varint’s length first (offset -= sovGenerated(v)), then writes forward within that reserved space. So the caller never has to know the varint’s length before making room — sovGenerated tells it.

Three tiers of marshaller

The serializer dispatches on how capable the object is. doEncode — used by the raw serializer — spells out the hierarchy:

// staging/src/k8s.io/apimachinery/pkg/runtime/serializer/protobuf/protobuf.go:505
	switch t := obj.(type) {
	case bufferedReverseMarshaller:
		// this path performs a single allocation during write only when the Allocator wasn't provided
		// it also requires the caller to implement the more efficient Size and MarshalToSizedBuffer methods
		if precomputedObjSize == nil {
			s := t.Size()
			precomputedObjSize = &s
		}
		data := memAlloc.Allocate(uint64(*precomputedObjSize))

		n, err := t.MarshalToSizedBuffer(data)
		if err != nil {
			return 0, err
		}
		return w.Write(data[:n])

	case bufferedMarshaller:
		// this path performs a single allocation during write only when the Allocator wasn't provided
		// it also requires the caller to implement the more efficient Size and MarshalTo methods
		if precomputedObjSize == nil {
			s := t.Size()
			precomputedObjSize = &s
		}
		data := memAlloc.Allocate(uint64(*precomputedObjSize))

		n, err := t.MarshalTo(data)
		if err != nil {
			return 0, err
		}
		return w.Write(data[:n])

	case unbufferedMarshaller:
		// this path performs extra allocations
		data, err := t.Marshal()
		if err != nil {
			return 0, err
		}
		return w.Write(data)

Three tiers: bufferedReverseMarshaller (best — marshals to the end of the buffer), bufferedMarshaller (good — marshals to the start, with the same single-allocation shape but a MarshalTo instead of a MarshalToSizedBuffer), and unbufferedMarshaller (the Marshal() ([]byte, error) interface, which allocates a fresh slice every time). CRDs written by third-party authors that use plain gogo/protobuf Marshal() land in the last tier and pay the copy.

Note the fallback in the memAlloc == nil branch: it logs an error and falls back to runtime.SimpleAllocator, which is make([]byte, n) — i.e. a fresh allocation every time. The SimpleAllocator is the “no pooling” mode.

The pooled allocator

One allocation per object is good. One allocation per response is better. That is what runtime.Allocator does:

// staging/src/k8s.io/apimachinery/pkg/runtime/allocator.go:116
func (a *Allocator) Allocate(n uint64) []byte {
	if uint64(cap(a.buf)) >= n {
		a.buf = a.buf[:n]
		return a.buf
	}
	// grow the buffer
	var size uint64
	if maxPool := maxPooledBufferCapacity.Load(); maxPool > 0 {
		// Grow geometrically up to maxPool (or n if a single request exceeds maxPool).
		size = max(n, min(uint64(2*cap(a.buf)), uint64(maxPool)))
	} else {
		// Although the legacy 2*cap+n formula over-allocates up to 3x, keep it
		// unchanged here to preserve exact prior behavior when the cap is disabled.
		size = uint64(2*cap(a.buf)) + n
	}
	a.buf = make([]byte, size)
	a.buf = a.buf[:n]
	return a.buf
}

Allocate(n) returns the same underlying array if it is already big enough. The only work per object is a capacity check and a slice reslice. If the buffer is too small it grows — and note it over-allocates (2*cap + n, up to 3x) so that the next object is likely to fit without another allocation.

The allocator itself is pooled:

// staging/src/k8s.io/apimachinery/pkg/runtime/allocator.go:38
var AllocatorPool = sync.Pool{
	New: func() interface{} {
		return &Allocator{}
	},
}

and the API server checks one out per response:

// staging/src/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:117
	var memoryAllocator runtime.MemoryAllocator
	if encoderWithAllocator, supportsAllocator := encoder.(runtime.EncoderWithAllocator); supportsAllocator {
		memoryAllocator = runtime.AllocatorPool.Get().(*runtime.Allocator)
		encoder = runtime.NewEncoderWithAllocator(encoderWithAllocator, memoryAllocator)
	}
	if memoryAllocator != nil {
		defer runtime.PutAllocator(memoryAllocator)
	}

The comment in the allocator source spells out the lifecycle precisely (staging/src/k8s.io/apimachinery/pkg/runtime/allocator.go:47): “In kube-apiserver, a single Allocator is checked out from AllocatorPool at the start of a response (SerializeObject for a GET or LIST) or watch stream (WatchServer.HandleHTTP / HandleWS) and reused across every object encoded during that LIST or WATCH before PutAllocator returns it to the pool when the request or watch stream ends.”

So a LIST of 10 000 Pods allocates one buffer, sized to the largest Pod, and reuses it 9 999 times. A watch stream does the same across every event for the life of the connection:

// staging/src/k8s.io/apiserver/pkg/endpoints/handlers/watch.go:135
	var memoryAllocator runtime.MemoryAllocator

	if encoderWithAllocator, supportsAllocator := negotiatedEncoder.(runtime.EncoderWithAllocator); supportsAllocator {
		// don't put the allocator inside the embeddedEncodeFn as that would allocate memory on every call.
		// instead, we allocate the buffer for the entire watch session and release it when we close the connection.
		memoryAllocator = runtime.AllocatorPool.Get().(*runtime.Allocator)
		negotiatedEncoder = runtime.NewEncoderWithAllocator(encoderWithAllocator, memoryAllocator)
	}

The bound on the pool. An unbounded pool has an obvious failure mode: a burst of large objects leaves every pooled allocator holding an oversized buffer forever. The allocator source names this explicitly — “Without a pool bound, an Allocator’s buffer never shrinks across requests, so a burst of large objects leaves every pooled allocator holding an oversized buffer for the lifetime of the process.”

So the bound is opt-in and process-wide, configured at startup with runtime.SetMaxPooledBufferCapacity(n); zero means unbounded, which is the default so that library consumers see no change. Note the deliberate asymmetry: an oversized buffer is still reused for the rest of the current LIST or WATCH — the bound only decides whether it goes back into the pool at the end.

Trade-offs

• The +8 varint over-allocation. estimateUnknownSize reserves 8 bytes for the length prefix when a varint may be at most 5 for realistic sizes. That is up to 7 wasted bytes per object, in exchange for not having to compute the exact varint length before allocating.
• Size must be computed. Size() walks the entire object graph. The reverse marshal then walks it again. The naive path walks it once but copies the payload twice. For large objects the reverse path wins on memory bandwidth; for tiny objects the extra walk is noise.
• CRDs pay the third-tier price. A third-party CRD that only implements Marshal() ([]byte, error) gets the unbufferedMarshaller path with its extra allocations and the extra unk.Size() walk.
• The pooled buffer is not zeroed. The allocator comment says so explicitly. The marshal writes every byte it returns, so this is safe here, but it is the kind of contract that bites when reused elsewhere.

Wrapping up

The one thing to remember: the Kubernetes protobuf wire format is a 4-byte “k8s” magic prefix plus a runtime.Unknown envelope, and the object is marshalled backwards from the end of a pre-sized buffer by Unknown.NestedMarshalTo (staging/src/k8s.io/apimachinery/pkg/runtime/types_proto.go:160). The object’s bytes are written directly into their final position — they are never copied.

The part people get wrong: that the payload is a plain protobuf encoding of the object. It is not. The object is a field (rawTag = 0x12) inside a generic envelope that also carries the TypeMeta, the content type and the content encoding. If you are writing a decoder, you must parse the envelope first — and check the magic prefix, because the API server uses it to pick a serializer.

What to go do about it: if you write a high-throughput client, ask for application/vnd.kubernetes.protobuf explicitly in your Accept header — it is the difference between the unbufferedMarshaller tier and the bufferedReverseMarshaller tier on the server. If you maintain a CRD, regenerate it with the Kubernetes code-generator so it implements Size()/MarshalToSizedBuffer and lands on the fast path. If you are chasing API server GC pressure, look at apiserver_runtime_malloc_* and the response size histograms: the runtime.Allocator pool is what keeps a LIST from allocating once per object, and runtime.SetMaxPooledBufferCapacity is the knob that stops a burst of large objects from pinning oversized buffers for the life of the process.