Children Of The Gods Of Old

Posted on Jun 30, 2021

I’ve been learning about the termios interface… You may know it as a header that allows you to do stuff like switching terminal echo’ing off (for password input, for example) and you may also have heard that it’s arcane as hell, so much that people avoid it like the plague to manipulate it directly. I find it a fascinating piece of software and software history, though… and I think it shows quite some facts that are still relevant today.

So, termios was born as an I/O API to program terminals… and terminals, back in the day, were actual devices that were linked to the computer via the serial port. That’s why the POSIX specs (yes, termios.h is part of POSIX!) talk about setting up the baud rates and why timing is critical to detect whether the user by pressing ESC and then k meant an escaped k keypress or two distinct keypresses. That’s why select(3) requires a time window as well. To use termios means to program hardware. That’s why you may still find tutorials that refer to Terminal Programming (i.e. using termios) indistinctively as Serial Programming as well!

Yep, but in 2021 nobody is using hardware VT-100 terminals… We’re all using terminal… emulators… Yes, that’s right, that’s where the emulator part comes from! Your favorite terminal emulator is exactly that: a piece of software that is emulating a terminal as those available in the old days.

So, why am I talking about this? Is this girl gone crazy and wants to talk about technology of the late 70s and the 80s? Well, yes, sort of…

So, What Was I Trying To Do?

I am working on implementing a history feature for scalc. By “history feature” I mean a totally automatic feature we’re so used to in our shells: press Up Arrow and you are able to navigate through previous commands… use Down Arrow to navigate to more recent commands.

That functionality is nowhere to be found in the C Standard Library. If you use any subroutine that reads a buffer from stdin and hit any arrow key… you’ll get some weird escape code:

/* arrowdemo.c */

#include <stdio.h>
#include <string.h>

int
main(void)
{
	char buf[256]; /* 256 looks like a nice number */

	/* memset()'ing so the strncmp() call is meaningful */
	memset(buf, 0, sizeof(buf));

	while(strncmp(buf, "bye", sizeof(buf)) != 0) {
		fgets(buf, sizeof(buf), stdin);
		buf[strlen(buf) - 1] = '\0'; /* Chomp '\n' */
	} 

	return 0;
}

Enjoy the utter failure of this program!

$ gcc -std=c99 -D_POSIX_C_SOURCE=200809L -Wall -Wextra -Wpedantic -o arrowdemo arrowdemo.c
$ ./arrowdemo
ohno^[[A^[[B^[[D^[[C
bye

Those escape codes are the arrow keys being pressed. And if you think you might just treat them as regular characters, I got some bad news for you: they’re terminal specific… and don’t fit well in a char variable:

/* arrowdemo2.c */

#include <stdio.h>
#include <string.h>

int
main(void)
{
	char buf[256]; /* 256 looks like a nice number */

	/* memset()'ing so the strncmp() call is meaningful */
	memset(buf, 0, sizeof(buf));

	while(strncmp(buf, "bye", sizeof(buf)) != 0) {
		fgets(buf, sizeof(buf), stdin);
		buf[strlen(buf) - 1] = '\0'; /* Chomp '\n' */
		if (buf[0] == '^[[A') /* Oh the ingenuity */
			printf("Up arrow!\n");
	} 

	return 0;
}

GCC complains and is totally right in doing so… and the whole thing doesn’t work either, it never recognizes that escape sequence as equivalent to the Up Arrow keypress even if we chomp the newline out of it.

$ gcc -std=c99 -D_POSIX_C_SOURCE=200809L -Wall -Wextra -Wpedantic -o arrowdemo2 arrowdemo2.c
arrowdemo2.c: In function 'main':
arrowdemo2.c:15:31: warning: multi-character character constant [-Wmultichar]
   15 |                 if (buf[0] == '^[[A') /* Oh the ingenuity */
      |                               ^~~~~~
arrowdemo2.c:15:28: warning: comparison is always false due to limited range of data type [-Wtype-limits]
   15 |                 if (buf[0] == '^[[A') /* Oh the ingenuity */
      |                            ^~
$ ./arrowdemo2
^[[A
nope, doesn't work
bye

No, trying to force this as a string comparison against “^[[A” doesn’t work either:

/* arrowdemo3.c - I promise, this is the last one */

#include <stdio.h>
#include <string.h>

int
main(void)
{
	char buf[256]; /* 256 looks like a nice number */

	/* memset()'ing so the strncmp() call is meaningful */
	memset(buf, 0, sizeof(buf));

	while(strncmp(buf, "bye", sizeof(buf)) != 0) {
		fgets(buf, sizeof(buf), stdin);
		buf[strlen(buf) - 1] = '\0'; /* Chomp '\n' */
		if (strncmp(buf, "^[[A", sizeof(buf)) == 0) /* Nope... */
			printf("Up arrow!\n");
	} 

	return 0;
}

Same garbage output as before on hitting Up Arrow and no response again:

$ ./arrowdemo3
^[[A
bye

So, what’s going on here? Why a simple gesture like hitting arrow keys, an action we do hundred of times a day… is so complicated to parse on a terminal. Is this a C thing? Nope:

#!/usr/bin/env python

# I know, this is horrible Python
while input() != "bye":
	pass
$ chmod +x arrowdemo.py
$ ./arrowdemo.py
^[[A
bye

But… there are programs which definitely are able to parse arrow keys and perform actions on them, right? So, OK, the C Standard Library I/O subroutines don’t work for this… I mentioned termios at the beginning of this post, so you already know this has to do with termios and Terminal Programming… and yes, that’s right… But have you ever wondered exactly why?

What Is History Anyways?

Let’s think what the history feature really is about.

  1. You are at a prompt (e.g. your shell prompt). This prompt is the last line in the output. The cursor flashes after some fixed string (your PS1 if it’s your shell prompt) you can’t edit and you can’t move the cursor to. On top of the active line there’s the backlog.
  2. You hit the Up Arrow.
  3. Something retrieves the last command from some data structure in memory.
  4. That last command is pasted into the active line, even completely replacing anything you might’ve written there before hitting the Up Arrow.
  5. Repeatedly using the arrow keys changes the whole active line.

Numbers 4 and 5 mean that there’s some serious screen manipulation going on. You’re redrawing the line. Also, the fact that you can’t delete the prompt by hitting backspace should also tell you that this input prompt is doing some heavy interpretation before really getting the command you typed in… Wait? Is it even getting strings? When you hit any arrow key, actions are performed immediately! Is this reading stuff character by character? Like, testing what every single keypress is and acting on that? Yup, that’s what your terminal is doing for you.

Have you used any ncurses-based program or any program that implements some sort of TUI? Have you ever wondered how that even works? In a nutshell those programs deal with drawing onto the terminal in a very similar way as GUI programs that don’t use any widget-based library (i.e., instead of using Qt, GTK+, etc., they interface withh X11 or Wayland directly.) They take the terminal screen as their canvas onto “paint” output. The difference is that non-widget based GUIs paint pixels, whereas TUIs paint ASCII characters… because that’s what terminals understand.

Not All Teletypewriters Were Born Equal

A very, very common conceptual nightmare novice Linux users deal with is with the following concepts: tty, terminal (emulator), shell, command line (CLI). Bear with me, because this is super important to understand the history of all of this.

  1. A terminal was literally a device, as I explained above. Nowadays, we have programs that emulate them. These devices were just… a place where to send output to and show a user (probably sitting in a different room than the one the actual computer was in) prompts and what they were typing in.
  2. A shell is a REPL: read, eval(uate), print loop. It takes user input, does something with it, prints the result… and starts again. It’s an interpreter… which you usually run by having a terminal deal with its I/O.
  3. Command line (interface) means that a program accepts user interaction via a text invocation… usually from a shell… but not necessarily: you can run a program using its CLI from a different program (exec & fork.)
  4. A tty is…

The tty acronym comes from teletypewriter. Teletypewriters predate computers: they were typewriters that were connected via the phone line and allowed sending text from one end to the paper feed on the other end.

Paper is the key here. UNIX was born in 1969/1970. Paper interfaces are not a thing nowadays, but totally were back then. A “paper interface” could mean a strip of paper where you could read the results of some calculation. Think of old business calculators that printed out results on their paper roll. Precisely that.

Our good ol' printf() subroutine is called print because of this. Because back then you couldn’t assume people had a monitor as their output device! The technology had to support both: printing to paper and “printing” to screen (usually a terminal.)

That’s why we’ve got the seemingly absurd “Carriage Return” (CR, ‘\n’, 0x0a) and “Line Feed” (LF, ‘\r’, 0x0d) ASCII characters. If you’ve ever played around ‘\r’, you surely know that it literally just places the cursor at the start of the line and new output will only overwrite the old output character by character, exactly like a typewriter would (just without the ink mess.) These characters came from the typewriting days… and were naturally adopted in computers because computers were still using paper as their interface.

You surely know the meme. “Ed is the standard UNIX editor!” You surely have gone through this as well:

$ ed
what am i supposed to do here?
?
heeelp
?
ohmy
?
ohno
?
q

UNIX ed is a line-based editor precisely because of the limitations regarding I/O back then. Look at this proper usage of Ed1:

$ ed
i
This is insert mode (sounds familiar??)
We're feeding lines into ed and this are a typo
.
,l
This is insert mode (sounds familiar??)
We're feeding lines into ed and this are a typo
s/are/is/
,l
This is insert mode (sounds familiar??)
We're feeding lines into ed and this is a typo
w test.txt
87
q

Don’t some things feel familiar? Sure they do. What’s that i command? Ain’t it like vi’s command to get into insert mode? Ed also has a for appending, for example… and what about s/are/is/ Yes, you’re right! Where do you think sED came from? Why does sed operate on lines? Where does the name vi come from, but visual (editor)? Because vi was optimized for… screens!

See my point? I guess you are, but let’s jump to a different place now…

Things changed over the years. In the late 70s, using a monitor was the norm. So, some computers did change some things…

“Are You Keeping Up With The Commodore?”

The video spot is amazing. I haven’t touched a Commodore computer in ages; I guess my last time I was 10-ish. The Commodore PET was very limited, but the revolution came with both the VIC-20 and, of course, the venerable Commodore 64.

The VIC-chip based Commodore computers (VIC-20 and C64, that is) had very advanced editing features for that time: the way the BASIC editor worked on those computers doesn’t differ at all from a modern text editor. If you move the cursor to a previous line and modify it, it’s modified on memory as well (as long as you keep the same line number.) The ZX Spectrum didn’t have that: On the Speccy you always typed into a prompt and your code was appended to the buffer… you couldn’t “navigate” through code and modify in situ; if you wanted to change anything, you had to rewrite the whole new line.

The magic was in the video interface chip, the VIC chip. In fact, video RAM addresses were easily referable via pokes and peeks.

Because text isn’t just text. Text on a screen is also a kind of video mode… just the simplest one. As soon as you want to break from a paper-like I/O like the one provided by the C Standard Library, you’re forced to deal with video… not in the SDL or framebuffer sense of it… but in the sense that you must instruct your video output device where to put things, what to replace, what not to replace, etc.

Ever written anything with ncurses? Cumbersome, isn’t it? That’s why. It’s text, but it’s actually a special video mode… That’s why you’re forced to tell it a lot of information on foreground, background colors, positioning, why y, x coordinates are represented as in video (0,0 on the top left corner), etc. ncurses just wraps termios in a way that is more portable across terminals… termios is just a way to program a termios-compatible terminal, be it a real old 70s terminal or your emulator.

Your terminal emulator is also a video mode. Even if it’s the latest version of your favorite FOSS terminal emulator, yours also thinks in terms of an old terminal screen of the late 70s, early 80s.

We Are The Children Of The Gods Of Old

Our current technology has a direct link to the olden days in which very crazy geniuses revolutionized life forever. Our hip, modern, high-level APIs might hide the details a bit, but the old foundations always show at some point.

When you look at the POSIX specs, you see this. How many parts are straight up taken from 70s UNIX. Of course modern POSIX has departed from “Research UNIX” and I’d be very surprised if a POSIX.1-2008-compliant binary worked on an old UNIX system. I wonder how far back we could go though?2

We are using those technologies, though. We might not be fully aware of this because the current implementations are certainly new. The definitions, however, were long defined in the past.

If you’re a science person you know this very well. “Dwarfs standing on the shoulders of giants,” right? This is to be expected. Radical changes in tech are uncommon; incremental changes are the norm. Of course nobody in the early days of home computing was in a position to predict our current Internet, but… hey… The Apple ][ and the Commodore 64 had modems and networking software available… AOL was the continuation of the C64 QuantumLink software. It wasn’t the modern Internet, let alone the modern web, and they didn’t use TCP/IP… but… some fundamentals were already available back then, as they had been already in UNIX local networks even before the 8-bits era.

There are millions of examples; termios is just one of them. That’s why its manpage still tells you about baud rates. That’s why coding the most hip language will always feel that you’re somehow dependent on how things are done under the hood, even if people tell you that high level languages save you from those details. They don’t, at least not fully.

The laws of computing and programming, the playing field, all of it was set long, long time ago… Besides very welcome hardware improvements, how many things are really new? Not that many. And you know what… I find that gives us reassurance and stability… to keep growing on solid ground.


  1. Wow, I just learned some Ed just for this post. This is commitment! ↩︎

  2. Of course, statically linking libraries… Trying this experiment using dynamic libraries probably won’t work on systems that are just a couple of years old. And of course we should also take into account CPU architectures. ↩︎