Opensource internals

tmux Runs Real ptys for Every Pane

9 min read Jay Hubbert

Every pane in tmux is backed by a real pseudo-terminal with a real session leader behind it, which is why job control, curses programs and mouse reporting work. It also means tmux has to deal with resize storms and the SIGTTIN problem.

TL;DR: A pane isn’t a picture of a terminal — it’s a real pty with a real process behind it, in its own session. That’s why vim and htop work inside tmux, and it’s also why tmux has to handle job control, resize storms and the SIGTTIN hazard that a screen-scraper never sees.

tmux doesn’t “draw” panes. Every pane is a real pseudo-terminal with a real process behind it, running in its own session with the pty as its controlling terminal.

That choice is what makes vim, htop, less, top and even tmux itself work inside a pane. It’s the difference between tmux and a screen-scraper.

It also creates a set of problems a screen-scraper doesn’t have: job control, resize storms, and the classic “background process tried to read from the terminal” problem.

Creating the pty

The pane’s process is created with forkpty(3):

c 
	/* Fork the new process. */
	new_wp->pid = fdforkpty(ptm_fd, &new_wp->fd, new_wp->tty, NULL, &ws);
	if (new_wp->pid == -1) {
		xasprintf(cause, "fork failed: %s", strerror(errno));
		new_wp->fd = -1;
		...
	}

fdforkpty() is a thin wrapper in compat/fdforkpty.c:

c 
int
getptmfd(void)
{
	return (INT_MAX);
}

pid_t
fdforkpty(__unused int ptmfd, int *master, char *name, struct termios *tio,
    struct winsize *ws)
{
	return (forkpty(master, name, tio, ws));
}

The ptmfd parameter exists for platforms without forkpty() — AIX, Haiku, HP-UX and Solaris have their own implementations in compat/forkpty-*.c that do the grantpt/unlockpt/ptsname dance by hand.

The winsize passed to forkpty() sets the initial terminal size, which means the program starts out with the right idea of how big it is:

c 
	/* Initialize the window size. */
	memset(&ws, 0, sizeof ws);
	ws.ws_col = screen_size_x(&new_wp->base);
	ws.ws_row = screen_size_y(&new_wp->base);
	ws.ws_xpixel = w->xpixel * ws.ws_col;
	ws.ws_ypixel = w->ypixel * ws.ws_row;

forkpty() also calls setsid() in the child and makes the slave pty its controlling terminal. That’s the important part: the child is a session leader, and the pty is its terminal.

Why that matters: job control

Because the pane’s process group is the foreground process group of its pty, the kernel delivers Ctrl-C, Ctrl-\ and Ctrl-Z to it exactly as it would on a normal terminal. tcgetpgrp() inside a pane returns the pane’s own process group, and /proc/<pgrp>/cwd gives the pane’s working directory — which is how tmux implements “current path” tracking:

c 
	if ((pgrp = tcgetpgrp(fd)) == -1)
		return (NULL);
	xasprintf(&path, "/proc/%lld/cmdline", (long long) pgrp);

A screen-scraper can’t do any of this, because there’s no pty — the program is writing to a pipe and the multiplexer has to pretend.

Gotcha: if you ever see a pane’s #{pane_current_path} disagree with what you’d expect, this is the machinery behind it. It reads the real kernel state of the real process group.

The SIGTTIN problem

Job control brings a specific hazard: a background process that reads from its terminal gets stopped by the kernel with SIGTTIN. In a normal shell that’s desirable. In a multiplexer it’s a bug — the program was never in the background from its own point of view.

tmux handles this in two places.

First, the server ignores the job control signals entirely:

c 
void
proc_set_signals(struct tmuxproc *tp, void (*signalcb)(int))
{
	struct sigaction	sa;

	tp->signalcb = signalcb;

	memset(&sa, 0, sizeof sa);
	sigemptyset(&sa.sa_mask);
	sa.sa_flags = SA_RESTART;
	sa.sa_handler = SIG_IGN;

	sigaction(SIGPIPE, &sa, NULL);
	sigaction(SIGTSTP, &sa, NULL);
	sigaction(SIGTTIN, &sa, NULL);
	sigaction(SIGTTOU, &sa, NULL);
	sigaction(SIGQUIT, &sa, NULL);

Note that tmux never calls tcsetpgrp() or TIOCSPGRP on the pane’s pty. It doesn’t need to: the pane’s child is already the session leader and the foreground process group. tmux isn’t part of that pty’s job control at all.

Second, when a child is stopped by SIGTTIN or SIGTTOU, tmux doesn’t treat it as a crash and doesn’t resume it:

c 
static void
server_child_stopped(pid_t pid, int status)
{
	struct window		*w;
	struct window_pane	*wp;

	if (WSTOPSIG(status) == SIGTTIN || WSTOPSIG(status) == SIGTTOU)
		return;

	RB_FOREACH(w, windows, &windows) {
		TAILQ_FOREACH(wp, &w->panes, entry) {
			if (wp->pid == pid) {
				if (killpg(pid, SIGCONT) != 0)
					kill(pid, SIGCONT);
			}
		}
	}

Any other stop signal gets a SIGCONT, because a process that stopped itself (or was stopped by SIGSTOP/SIGTSTP) would otherwise stay stopped forever with nobody to type fg.

The outer terminal is a different story

The server writes to the client’s terminal, and it isn’t the foreground process group of that terminal — the client’s shell is. That’s safe because TOSTOP is off by default on most terminals, and tmux never leaves the client’s foreground group (the client owns the tty).

When the user wants to give the terminal back to their shell — Ctrl-Z from inside tmux — it’s the client that does it, not the server:

c 
	sigact.sa_handler = SIG_DFL;
	if (sigaction(SIGTSTP, &sigact, NULL) != 0)
		fatal("sigaction failed");
	client_suspended = 1;
	kill(getpid(), SIGTSTP);

and on SIGCONT the client tells the server to wake up and redraw.

Resizing: the storm problem

Resizing a pane means calling TIOCSWINSZ on the pty master, which makes the kernel send SIGWINCH to the foreground process group:

c 
void
window_pane_send_resize(struct window_pane *wp, u_int sx, u_int sy)
{
	struct window	*w = wp->window;
	struct winsize	 ws;

	if (wp->fd == -1)
		return;

	log_debug("%s: %%%u resize to %u,%u", __func__, wp->id, sx, sy);

	memset(&ws, 0, sizeof ws);
	ws.ws_col = sx;
	ws.ws_row = sy;
	ws.ws_xpixel = w->xpixel * ws.ws_col;
	ws.ws_ypixel = w->ypixel * ws.ws_row;
	if (ioctl(wp->fd, TIOCSWINSZ, &ws) == -1)
#ifdef __sun
		/*
		 * Some versions of Solaris apparently can return an error when
		 * resizing; don't know why this happens, can't reproduce on
		 * other platforms and ignoring it doesn't seem to cause any
		 * issues.
		 */
		if (errno != EINVAL && errno != ENXIO)
#endif
		fatal("ioctl failed");
}

The Solaris comment is a perfect example of the kind of thing that accumulates: an errno filter added to work around a platform bug nobody can reproduce.

The problem with resizing is the storm. Dragging a terminal window edge produces dozens of resize events per second. If each one became a TIOCSWINSZ and a SIGWINCH, the program inside the pane would redraw dozens of times per second, burning CPU and flickering.

tmux solves this with a resize queue and a timer. The queue holds pending resizes; a timer (250 ms) fires and decides what to actually do. The decision logic is documented in the source:

c 
	/*
	 * There are three cases that matter:
	 *
	 * - Only one resize. It can just be applied.
	 *
	 * - Multiple resizes and the ending size is different from the
	 *   starting size. We can discard all resizes except the most recent.
	 *
	 * - Multiple resizes and the ending size is the same as the starting
	 *   size. We must resize at least twice to force the application to
	 *   redraw. So apply the first and leave the last on the queue for
	 *   next time.
	 */

The third case is the subtle one. If the user drags the edge out and back to the original size, the net change is zero — and a single TIOCSWINSZ to the same size would produce no SIGWINCH, because the kernel only signals on an actual change. The application would never redraw, and the pane would be left in a stale state. The fix is to apply an intermediate size and then the final size, forcing two signals.

The implementation:

c 
	first = TAILQ_FIRST(&wp->resize_queue);
	last = TAILQ_LAST(&wp->resize_queue, window_pane_resizes);
	if (first == last) {
		/* Only one resize. */
		window_pane_send_resize(wp, first->sx, first->sy);
		TAILQ_REMOVE(&wp->resize_queue, first, entry);
		free(first);
	} else if (last->sx != first->osx || last->sy != first->osy) {
		/* Multiple resizes ending up with a different size. */
		window_pane_send_resize(wp, last->sx, last->sy);
		window_pane_clear_resizes(wp, NULL);
	} else {
		/*
		 * Multiple resizes ending up with the same size. There will
		 * not be more than one to the same size in succession so we
		 * can just use the last-but-one on the list and leave the last
		 * for later. We reduce the time until the next check to avoid
		 * a long delay between the resizes.
		 */
		r = TAILQ_PREV(last, window_pane_resizes, entry);
		window_pane_send_resize(wp, r->sx, r->sy);
		window_pane_clear_resizes(wp, last);
		tv.tv_usec = 10000;
	}

There’s also a special case: when a pane enters the alternate screen, any pending resize is flushed immediately rather than waiting for the timer, because the alternate screen is about to take over the display and the application will redraw anyway. There’s a regression test for exactly this (regress/alternate-screen-resize.sh).

Not blocking the application

The classic multiplexer failure mode: the application writes faster than the multiplexer drains the pty, the pty buffer fills, and the application blocks.

tmux avoids this by always draining the pty master into its own grid. The read is done by a libevent bufferevent, and the read event is explicitly disabled while parsing:

c 
	input_parse_pane(wp);
	bufferevent_disable(wp->event, EV_READ);

so that a program producing output at full speed can’t starve the rest of the event loop. The data is parsed and the read event re-armed afterwards.

The unbounded thing isn’t the pty buffer but tmux’s own scrollback, and that’s bounded by the history-limit option, trimmed 10% at a time (see How tmux Stores Grid Cells in Five Bytes).

The other unbounded thing is a hostile program sending an endless unterminated escape sequence. That’s bounded separately, by the input-buffer-size option, defaulting to 1 MB, after which the parser discards the sequence.

Background jobs without a pty

Not everything needs a terminal. run-shell and #() format jobs run without one, using a socketpair instead:

c 
	if (flags & JOB_PTY) {
		pid = fdforkpty(ptm_fd, &master, tty, NULL, &ws);
	} else {
		if (socketpair(AF_UNIX, SOCK_STREAM, PF_UNSPEC, out) != 0)
			goto fail;
		pid = fork();

and their output is captured rather than rendered. The distinction matters: a #() job in a status line must not be able to mess with the screen.

Wrapping up

Using real ptys is the decision that makes tmux a good terminal multiplexer rather than a screen-scraper. Everything else in this article is the cost of that decision:

  • You must implement a terminal emulator, because you now have to know what the program drew.
  • You must handle job control, because the kernel is now involved.
  • You must handle resize storms, because SIGWINCH is now delivered to a real process.
  • You must handle the pty buffer, because a real program is now writing to a real device.

Every one of those is a problem a screen-scraper simply doesn’t have — and every one of them shows up in a user’s terminal as “tmux is doing something weird again”.

TmuxArchitectureInternalsPty