tmux Batches Terminal Output: One Write() Per Event Loop Iteration

TL;DR: Every byte tmux sends to a terminal goes through one function that appends to a per-client buffer, and that buffer is flushed with a single write() per event loop iteration. If the terminal can’t keep up, tmux throws the output away and repaints the screen from scratch later — a deliberate trade of bytes for never blocking.

Every byte tmux sends to a terminal goes through one function. That function appends to a buffer, the buffer is written out with a single write() per event loop iteration, and if the terminal cannot keep up, the buffer is thrown away and the screen is redrawn from scratch later.

That is the whole output pipeline. And it’s a good example of a design where the priority is “the terminal must never be the thing that makes tmux slow” rather than “never lose a byte”.

One buffer per client

Each attached client has a struct tty, and each tty has an evbuffer called out. Everything goes through tty_add() (tty.c):

c
static void
tty_add(struct tty *tty, const char *buf, size_t len)
{
	struct client	*c = tty->client;

	if (tty->flags & TTY_BLOCK) {
		tty->discarded += len;
		return;
	}

	evbuffer_add(tty->out, buf, len);
	log_debug("%s: %.*s", c->name, (int)len, buf);
	c->written += len;

	if (tty_log_fd != -1)
		write(tty_log_fd, buf, len);
	if ((tty->flags & TTY_STARTED) &&
	    !event_pending(&tty->event_out, EV_WRITE, NULL))
		event_add(&tty->event_out, NULL);
}

Two details matter more than the rest.

The TTY_BLOCK check comes first. If the client is blocked, the bytes are counted in tty->discarded and dropped on the floor. They are not buffered, not queued, not anywhere.
The event_pending() check. Without it, every single escape sequence would try to arm the write event. Checking whether it is already armed turns “arm the event” from an operation per escape sequence into an operation per event loop iteration.

That second one is the kind of thing you only notice when you go looking. It’s also the difference between thousands of redundant event_add() calls per second and a handful.

One write() per iteration

The buffer is drained by a libevent callback:

c
static void
tty_write_callback(__unused int fd, __unused short events, void *data)
{
	struct tty	*tty = data;
	struct client	*c = tty->client;
	size_t		 size = EVBUFFER_LENGTH(tty->out);
	int		 nwrite;

	nwrite = evbuffer_write(tty->out, c->fd);
	if (nwrite == -1)
		return;
	log_debug("%s: wrote %d bytes (of %zu)", c->name, nwrite, size);

	if (c->redraw > 0) {
		if ((size_t)nwrite >= c->redraw)
			c->redraw = 0;
		else
			c->redraw -= nwrite;
		log_debug("%s: waiting for redraw, %zu bytes left", c->name,
		    c->redraw);
	} else if (tty_block_maybe(tty))
		return;

	if (EVBUFFER_LENGTH(tty->out) != 0)
		event_add(&tty->event_out, NULL);
}

evbuffer_write() drains as much as the kernel will take in one write() call. If the buffer is not empty afterwards, the event is re-armed and we go around again.

Here’s the payoff: redrawing a 200x50 window with four panes produces one write() syscall. Not one per cell, not one per run, not one per escape sequence. On a local terminal that is the difference between a smooth redraw and a visible stall.

There’s a second, subtler reason for buffering: ordering. tmux sends cursor moves, SGR changes, erases and text, and those have to arrive in order. If they were separate writes, a terminal that processed them out of order — or a write() that returned a partial count — would produce a garbled screen. One buffer, one write, one order.

The slow terminal problem

Now the part that makes this design interesting.

A terminal on a slow link (SSH over a bad connection, a terminal emulator busy doing something else) can be slower than the program producing output. If tmux simply buffered everything, the buffer would grow without bound and tmux would eat all the memory.

tmux’s answer is to throw the output away:

c
#define TTY_BLOCK_INTERVAL (100000 /* 100 milliseconds */)
#define TTY_BLOCK_START(tty) (1 + ((tty)->sx * (tty)->sy) * 8)
#define TTY_BLOCK_STOP(tty)  (1 + ((tty)->sx * (tty)->sy) / 8)

The threshold is eight screenfuls of pending output. Once the buffer exceeds that, tty_block_maybe() drains the entire buffer:

c
static int
tty_block_maybe(struct tty *tty)
{
	struct client	*c = tty->client;
	size_t		 size = EVBUFFER_LENGTH(tty->out);
	struct timeval	 tv = { .tv_usec = TTY_BLOCK_INTERVAL };

	if (size == 0)
		tty->flags &= ~TTY_NOBLOCK;
	else if (tty->flags & TTY_NOBLOCK)
		return (0);

	if (size < TTY_BLOCK_START(tty))
		return (0);

	if (tty->flags & TTY_BLOCK)
		return (1);
	tty->flags |= TTY_BLOCK;

	log_debug("%s: can't keep up, %zu discarded", c->name, size);

	evbuffer_drain(tty->out, size);
	c->discarded += size;
	...

The client is now flagged TTY_BLOCK, so every subsequent tty_add() is dropped. A timer runs every 100 ms and, once the buffer has drained below the stop threshold, the client is flagged for a full redraw:

#define TTY_BLOCK_STOP(tty)  (1 + ((tty)->sx * (tty)->sy) / 8)

So the recovery is: lose the intermediate output, but resynchronise the terminal with a complete repaint.

This is a deliberate trade. The alternative — buffering without limit — turns a slow network into a slow tmux and eventually into an OOM kill. The cost is that a user on a slow link sees the screen jump rather than scroll smoothly, and any program whose output matters (a log tail, an interactive REPL) loses the text that was produced while tmux was blocked.

The CHANGES file records the thinking:

“Handle slow terminals and fast output better: when the amount of data outstanding gets too large, discard output until it is drained and we are able to do a full redraw.”

Deferring redraws until the buffer drains

Related to the above: tmux does not redraw on top of pending output. If there’s still data in the buffer, the redraw is deferred:

c
	/* Ignore output queued within the current synchronized frame. */
	n = EVBUFFER_LENGTH(tty->out);
	if ((tty->flags & TTY_SYNCING) && n > tty->sync_offset)
		n = tty->sync_offset;

	/* Defer until output drains, preserving damage in client flags. */
	if (n != 0 || (tty->flags & TTY_BLOCK)) {
		if (n != 0)
			log_debug("%s: redraw deferred (%zu left)", c->name, n);
		else
			log_debug("%s: redraw deferred (blocked)", c->name);
		if (!evtimer_initialized(&ev))
			evtimer_set(&ev, server_client_redraw_timer, NULL);
		if (!evtimer_pending(&ev, NULL)) {
			log_debug("redraw timer started");
			evtimer_add(&ev, &tv);
		}
		if (damaged) {
			c->flags |= CLIENT_REDRAWWINDOW;
			return;
		}

This is in server_client_check_redraw(). A 1 ms timer is armed, and the damage information is preserved in the client’s flags so the deferred redraw still knows what changed. The damage is not thrown away and recomputed — it is carried across the deferral.

The reason is not just tidiness. Redrawing on top of pending output means the redraw bytes get interleaved with the pending bytes in the wrong order relative to what the terminal has already been told. You get tearing. Deferring keeps the byte stream consistent.

Caching the terminal’s attribute state

The second big source of wasted bytes is SGR. Every change of colour, attribute or underline style is an escape sequence, and a screen full of coloured text would otherwise be an enormous stream of ESC[...m.

tmux keeps the terminal’s current style in tty->cell and tty->last_cell, and refuses to emit a redundant change:

c
	/* Ignore cell if it is the same as the last one. */
	if (gc2.attr == tty->last_cell.attr &&
	    gc2.fg == tty->last_cell.fg &&
	    gc2.bg == tty->last_cell.bg &&
	    gc2.us == tty->last_cell.us &&
		gc2.link == tty->last_cell.link)
		return;

Same idea for the cursor, the scroll region and the terminal modes:

c
/* Move cursor to absolute position. */
void
tty_cursor(struct tty *tty, u_int cx, u_int cy)
{
	...
	/* No change. */
	if (cx == thisx && cy == thisy)
		return;

c
/* Set region at absolute position. */
static void
tty_region(struct tty *tty, u_int rupper, u_int rlower)
{
	if (tty->rlower == rlower && tty->rupper == rupper)
		return;
	if (!tty_term_has(tty->term, TTYC_CSR))
		return;

The cursor case has an extra wrinkle — the “automargin space”, the phantom column at the right edge of the line:

c
	/*
	 * If in the automargin space, and want to be there, do not move.
	 * Otherwise, force the cursor to be in range (and complain).
	 */
	if (cx == thisx && cy == thisy && cx == tty->sx)
		return;
	if (cx > tty->sx - 1) {
		log_debug("%s: x too big %u > %u", __func__, cx, tty->sx - 1);
		cx = tty->sx - 1;
	}

The cursor’s logical position can be one past the last column (the terminal is about to wrap), and tmux has to model that state to avoid emitting a cursor move that would reset the terminal’s wrap flag.

c
void
tty_update_mode(struct tty *tty, int mode)
{
	...
	if (tty->mode == mode)
		return;

The mode function also has a fix-up that shows how much terminal-specific knowledge is baked in:

c
	/*
	 * If the mouse modes have changed, clear then all and apply
	 * again. There are differences in how terminals track the
	 * various bits.
	 */
	tty_puts(tty, "\033[?1006l\033[?1000l\033[?1002l\033[?1003l");

Rather than trying to unset one mouse mode and set another, tmux clears all of them and re-applies the ones it wants. That’s more bytes in the common case, and fewer in the pathological case where a terminal tracks the bits differently.

Colours get the same treatment, with a diff:

c
	changed = gc2.attr & ~tc->attr;

so only the attribute bits that are actually new get emitted.

Terminfo strings are resolved once

Escape sequences come from terminfo, and terminfo strings contain parameters. tmux resolves them at startup rather than per use, and it prefers the allocating variant of the parameter expander:

c
const char *
tty_term_string_i(struct tty_term *term, enum tty_code_code code, int a)
{
	const char	*x = tty_term_string(term, code), *s;
#if defined(HAVE_TIPARM_S)
	s = tiparm_s(1, 0, x, a);
#elif defined(HAVE_TIPARM)
	s = tiparm(x, a);
#else
	s = tparm((char *)x, a, 0, 0, 0, 0, 0, 0, 0, 0);
#endif

tiparm_s is preferred because tiparm() and tparm() use a static buffer, which is both a thread-safety hazard and a source of surprises when two results are alive at once.

Terminfo padding ($<...>) is stripped at parse time, not at use time:

c
/*
 * Ignore strings with no padding.
 */

The raw mode handshake

Before any of this happens, the server takes the client’s terminal out of cooked mode. This is tty_start_tty() in tty.c, and it is a single tcsetattr():

c
	tio.c_iflag &= ~(IXON|IXOFF|ICRNL|INLCR|IGNCR|IMAXBEL|ISTRIP);
	tio.c_oflag &= ~(OPOST|ONLCR|OCRNL|ONLRET);
	tio.c_lflag &= ~(IEXTEN|ICANON|ECHO|ECHOE|ECHONL|ECHOCTL|ECHOPRT|ECHOKE|ISIG);

Two details matter for performance and correctness:

IXON/IXOFF are cleared, so Ctrl-S and Ctrl-Q reach the application instead of stopping the terminal’s output.
OPOST is cleared, so output is not translated — no \n → \r\n conversion, which means tmux does not have to think about the terminal eating a newline.

The fd is set non-blocking, so the event loop never blocks on a write:

c
	setblocking(c->fd, 0);

Putting the numbers together

For a typical frame — say, a status line update and a few changed cells in one pane — the output is:

text

1 × cursor position
1 × SGR change (maybe)
n × short text runs
1 × cursor position

all in one buffer, flushed with one write(). For a full-screen redraw of four panes it’s one buffer, one write(), and the byte count is proportional to the number of changed cells rather than the number of cells.

Wrapping up

The interesting decision here is not the buffering — that’s standard. It’s the willingness to discard output. tmux treats the terminal as a device that may be arbitrarily slow and decides that falling behind means resynchronise, not queue.

That decision only works because of the other half of the design: tmux keeps a complete model of what the screen should look like, so it can always rebuild it. Without that model, discarding output would mean losing the screen. With it, discarding output costs one full redraw.

Worth internalizing: “never lose a byte” and “never stall” are different goals, and a multiplexer that owns the screen model can safely pick the second one.