« How to play Diversity | Home

alden: detachable terminal sessions without breaking scrollback

Wed 11 Jun 2025 by mskala Tags used: , ,

For a long time I've wanted to have a thing for resuming interrupted ssh sessions. The typical scenario is that I connect to a server with ssh, start a long-running task, and leave the window open so I can watch it. Then something causes the connection to break. Maybe I lose my local networking connection; maybe there's a power failure at my end; maybe I need to reboot my computer for some reason. What happens to the task on the server side?

Very often, a connection loss means the task on the server side will receive a SIGHUP and automatically terminate. If I don't want that to happen, I can plan ahead with such tools as "nohup," "disown," output redirection, and "tail -f," to keep the task running regardless of a lost connection. But there are a lot of compromises involved in doing that. In particular, it doesn't work well for tasks that really are in some way interactive; even if I can keep the task running, I don't have much ability to control and interact with it, once it's been spun off into the server's subconscious.

There are tools that provide in different ways for reconnecting to a disconnected terminal session. Some of the best-known are mosh, tmux, and screen. But they all break scrollback, in the sense that while running them, if text scrolls off the top of the screen, you can't scroll back to that text using the native scrollback feature of your GUI terminal emulator. They. Break. Scrollback. They violate one of the most basic rules of user interface design, especially significant for Web pages but also important for terminal-based applications: never mess with the scrollbar.

The tmux and screen packages are said to provide scrollback, or even to "support" scrollback, but that is true only in a limited and unsatisfactory sense: mosh, tmux, and screen are basically all terminal emulators themselves, which run as an additional layer inside your GUI terminal emulator. In the cases of tmux and screen, they have scrollback features of their own. So by using a different set of commands unique to tmux or screen, not the scrollbar built into your GUI, you can get access to the history of scrolled-away text in tmux or screen. But it remains that the GUI's native scrollback is broken when you use any of these.

The mosh developers add insult to injury because their stock response to complaints about how mosh breaks scrollback, is to tell users to use tmux or screen inside mosh. They have been taking that position since 2012.

Part of the problem there is that mosh's protocol is based on an abstraction that is not easy to reconcile with scrollback. Instead of conveying a stream of characters that resumes after interruption, mosh's protocol is based on the idea of keeping variables synchronized between the two ends of the connection. The distinction is important in computer-science terms, and necessary for mosh to do the things it does do that its creators consider high-priority. One advantage to the variable-synchronization approach is that it means the server does not need to keep arbitrarily large buffers of past history that the client may have missed; it only needs to keep the latest state of the variables, and allow the client to query back any parts that are not up to date.

But it's hard to even state the scrollback problem clearly in the language of keeping variables synchronized, let alone actually solve it. Proposals have included defining a "virtual" screen that is something like three times the height of the real screen, and keeping that synchronized in a place that the native scroll bars can reach - as if scrollback users would be satisfied with just two pages' worth of previous history. Having a system that allows scrollback to really work, pretty much requires returning to the abstraction of a stream of characters.

It's been 13 years, and this week I finally got sufficiently annoyed to sit down and write the utility I actually want: a thing for resuming terminal connections that does not act as a terminal emulator itself, instead passing traffic through to the native terminal so that scrollback won't break.

I call the result "alden," and you can download the latest source release here:

It's GPL3. It is, obviously, a work in progress. It works pretty well for my purposes already, but I only have bandwidth to add extra features and improve portability, if other people want to use it, and especially if other people want to write about it.

After building, run the "alden" command-line program to start a new shell, which will be wrapped inside a client and server. If you lose your network connection or the client dies for some other reason, the server and shell ought to survive. Then you can log back in again, type "alden" again, and be connected to the existing server and shell. Further details are in the man page inside the package; even further details will probably be in other documentation if and when I do another release.

Unlike mosh, this program does not implement a high-reliability networking protocol of its own. The only IPC it does is through local mechanisms like named pipes. You still have to create an ssh connection and then run alden inside your ssh connection, and on a really bad network with a lot of packet loss, it works no better than ssh does. The thing it's intended to protect against is, instead, the situation where a basically usable network connection is suddenly lost entirely.

By design, alden also does not interpret or translate terminal command sequences in any way, in either direction. The closest it comes is that it passes through SIGWINCH terminal-size updates, so if you resize your window, aware programs inside the alden session ought to be able to detect that and adjust themselves accordingly.

NEW: version 0.2!

Thank you to everyone who has downloaded this and reported their experiences. I've posted a new version at the link above. In the new version:

16 comments

*
how do i "detach" the client from the server? i tried the following:

```
# terminal 1
$ ./alden
$ pstree -s -p $$ -l
systemd(1)───systemd(1109599)───x-terminal-emul(1110718)───zsh(1179908)───alden(1181725)───alden(1181726)───zsh(1181727)───pstree(1184549)

# terminal 2
$ kill -SIGHUP 1181726 # no observable change to terminal 1.
$ ./alden -r
No server.
```

i tried several other signals as well; all either had no effect or caused the server to exit along with the client.
jyn - 2025-06-16 06:56
*
To deliberately detach the session, kill the client, not the server. I believe in your example that would be process 1181725. The point here is for the server to survive the loss of the client; the server is otherwise a normal process, which can be killed with -HUP and isn't designed to preserve itself against signals in general.

You also might want to try just closing the window, if you're using a GUI terminal emulator - that could better simulate a network disconnect kind of situation.

Deliberately detaching the session is not really what I imagined as the intended use case, but it is useful for testing at least. Do you think it'd be valuable to have a feature built into the command-line program for doing this?
Matthew Skala - 2025-06-16 07:09
*
Further to that - I should read my own code. The server actually does ignore -HUP, hence your observation of "no change" on sending -HUP to the server. It remains that the client, not the server, is the right place to send the signal.
Matthew Skala - 2025-06-16 07:22
*
oh! i hadn't realized the parent process was the client and the child was the server, i'm used to it being the other way around. yes, sending HUP to the parent worked like a charm.

note for myself: when reconnecting the second time, the client PID is *not* in pstree, i suppose because it's only streaming output from the server and isn't actually the parent that spawned the shell. in that case the only way i've found to get the pid is to use `alden -v` on startup.

now that i'm able to consistently reattach, i can see that alden works differently than i imagined: it doesn't *restore* the scrollback when you reattach, it only prints the output that was emitted since you detached. that makes sense, but it means it's harder to do the thing i wanted, which is to build tmux's session persistence on top of alden and kitty's multiplexing support. i'm imagining how this could possible work. https://github.com/kovidgoyal/kitty/issues/2454 shows how to save and restore the native scrollback, and i should be able to get the server's PID, which is unique per-machine—i think that's enough for me to build a save/restore mechanism on top. and then i can combine that with https://sw.kovidgoyal.net/kitty/launch/#watching-launched-windows to save the scrollback on pane close and restore it when reattaching to alden.

if you could add built-in commands to get the pid of the current client and server, and a command to detach the current client, that would be very helpful. i think everything else is possible from there, and becomes my problem and not yours.

thank you for your work on this!
jyn - 2025-06-16 19:19
*
oh, one more thing that would be very helpful: if there were an option to `tee` the terminal output not just to the native terminal emulator but also to a file; and furthermore to write all output to that file even if no client is currently connected. i don't think there's any way to build that on top, because right now output gets discarded if a client is detached. the separate file should spare you from needing to buffer anything from memory, so hopefully it won't be too complicated?
jyn - 2025-06-16 19:25
*
I'll send you an email at the address you entered with your comments - that'll probably be an easier way to discuss technical details than this comment thread.
Matthew Skala - 2025-06-17 05:47
*
How does this differ from eternal terminal?
Foobar - 2025-06-23 10:40
*
I started trying to compile Eternal Terminal from a Git checkout and gave up when the directory tree, with the dependencies it wanted to download and install, reached 2.6 gigabytes. It didn't seem to be worth the amount of work that would be involved to press further, especially when it wasn't (and still isn't) clear to me that it *really* doesn't break scrollback in the first place. My own code is much, much smaller.
Matthew Skala - 2025-06-23 11:11
*
I'll probably use this in the context of a bunch of iTerm panes all SSHing to the same host. Is there a way to control which server the client connects to? Currently it looks like if multiple servers are running, the client chooses arbitrarily.
Evan - 2025-06-23 15:38
*
Planned for next version. It opens a can of worms because then the next question is how to figure out which server actually is which, but just having an option for the PID of the server is easy enough.
Matthew Skala - 2025-06-23 15:57
*
Any plans to host the source code somewhere?
Nikolay Kolev - 2025-06-23 17:34
*
That's what the tarball is.

I'm not planning to make my version control repository public, if that's what you mean.
Matthew Skala - 2025-06-23 17:43
*
This is great, thanks! Since you mentioned portability -- I'm on macOS so I grabbed the tarball and figured out how to get it to build and, eventually, run. There were some trivial platform differences to compensate for (e.g. macOS lacks pipe2, but since you pass 0 as your third argument your pipe2 invocation is equivalent to a bare call to pipe) but there was one deeply annoying fundamental issue -- on macOS, contrary to all indications in the documentation, you cannot use advisory locks on named pipes! By the documentation, your locking scheme should work on all platforms, and yet...

So adding macOS support would require you to change to your pipe locking scheme from the elegant way you do things to something like using an auxilary lockfile next to the pipes. I'm not sure whether you have any interest in macOS support, but in case you are interested, I'm happy to put together a patch and send it in for you.
Josh Kopin - 2025-06-24 07:17
*
Thanks for the report. It wouldn't be a disaster to create yet another file in /tmp just for locking, on platforms that need it. Do you know if there's an Autoconf test for this situation (of flock not working on pipes), other than just trying to detect MacOS and assume that it applies if running on MacOS?
Matthew Skala - 2025-06-24 07:25
*
Putty has a "reconnect" (menu) option that also preserves history.
ReD - 2025-06-24 19:50
*
Not really comparable. PuTTY is a GUI terminal emulator. It's not something I can run in my existing terminal, and although it *provides* a scrollback feature of its own (as do tmux and screen) it doesn't *not break* the scrollback of the existing terminal. It also appears that you can only "reconnect" from the same client machine, and only if the client machine hasn't been rebooted and maybe even only if the window hasn't been closed. If the loss of connection was such that the server-side process received a SIGHUP - which is a common case - then the long-running task we were trying to preserve will already be dead before you can use PuTTY reconnect; and output on the server side isn't going to be buffered (at best, the task will wait) while no client is connected. So PuTTY's feature, although no doubt useful to the people who use it, solves a completely different set of problems.
Matthew Skala - 2025-06-25 05:30


(optional field)
(optional field)
Answer "bonobo" here to fight spam. ここに「bonobo」を答えてください。ＳＰＡＭを退治しましょう！
I reserve the right to delete or edit comments in any way and for any reason. New comments are held for a period of time before being shown to other users.