The APIs of Executables

Posted on Aug 8, 2021

When we speak APIs, we usually speak libraries, right? Or maybe the kernel syscalls, but usually we don’t think of executables having any obvious APIs. This unawareness is sad, because the UNIX Philosophy sort of defines a default framework for defining APIs for executables, and actually you use them quite a lot… even if you don’t think of them as APIs in the library sense of it. So, as today is Sunday and I’m in the mood for some true geekness, let’s uncover one of the most important and possibly lesser known key concepts of UNIX, Unix-y OSs, POSIX, or whatever you call what we use!

What is an API? I like to think of them as the linguist I am, in fact. My way to define them is as: “A way for a human to make code speak with other code.” Lockean Linguistic Turn, anyone? That’s how I like conceptualizing things.

The human component in this is important: it differentiates between an ABI from an API. ABIs are similar, but also different, and can be independent from APIs as well… an API change might not entail an ABI change, e.g. if you change the signature of a subroutine from, let’s say, int64_t func(int64_t); to size_t func(size_t);. In many platforms both data types will be the same at the CPU level… in others they won’t. So the ABI might change or not depending on the platform, but the API has changed for all developers using that subroutine… and they’ll probably be very angry at you for a change like that, by the way.

The other part of my Totally Invented at Home definition of an API is that it’s about code speaking to other code. A GUI usually does not offer an API, unless you’re somehow able to programatically direct it… à la Squeak or other visually based programming platforms. But if we’re talking our usual suspects, APIs are textual… Be them a C header file, which is a text file, or a JSON stream you send via curl, or anything in between.

One of the traditional UNIX tenets is to “Write programs to handle text streams, because that is a universal interface.”1 And that’s one of the API entry points you get in programs written following the UNIX philosophy: text streams, whether they’re files on a disk or stdin/stdout. Your program expects that text stream to conform to some format and is expected to output in some defined format. If you do so, you ensure the mythical interoperability UNIX is famous for…

…And that’s why it’s an API, even if text streams are also user interfaces (e.g. you read user input from stdin), because… shell scripts!

If your script reads some data from stdin, be it from user input or from the output of another executable (which may be a native binary, something in an interpreted language, or another script… who cares!), it’s probably expecting that data to have some format… or at least it will make some assumptions about what that data looks like, e.g. is it a date, a string, a list of tab-separated value, a set of commands the script is able to understand, etc. That already constitutes an API entry point if you think about it.

The same goes for output, of course… especially in the Unix-y way of doing things you never know where your output will end up going to. Every terminal program that prints things to stdout will eventually be part of a pipeline, trust me. So you better treat your output as an API as well, as the return value of a C subroutine, if you will… Because someone will count on it in a pipeline sometime, somewhere… That someone might even be yourself in the future!

And don’t make me get into what happens when you’re using file redirection magic… See? In a POSIX system there is no good in assuming stdin, stdout, stderr are just the terminal… My beloved isatty() is a tool I always rely on to make sure what the pipe is going on!

But there’s another way in which programs interface with other programs: their CLI and exit values, to a lesser degree. Yes, true, the latter are usually typed in by the user in their lovely terminal emulators… but again, scripts… or hey, the Standard C system() function or any of the exec*() family of POSIX functions in C… I mean, these might be frowned upon because yes, it’s better to rely on the library rather than the executable, whenever is possible… but these also exist and have their use.

If an option changes its behavior or suddenly is deprecated… how different would that be to a library changing the types of a parameter in a function or changing their order? Your scripts relying on that specific CLI would start to fail and I’m pretty sure you’d be sending an angry email to the devs of the executable asking them why they did that…

I’m starting to think that I’m defining APIs to be “Things that, if changed, will make you get angry emails.” Hey, it’s not a bad definition!

So… why am I bothering to tell you things that you probably already know?

Why Sometimes It’s Just a Revision Release…

There’s this thing called Semantic Versioning. In a nutshell, SemVer works like this. For a version number X.Y.X,2

  1. The major number X means public API version. It should only change when the API has become backwards incompatible with the previous version’s.
  2. The minor/point number Y should only change when there have been backwards compatible changes to the API.
  3. The revision/patch number Z should change in cases where no API change has occured; usually, bugfixes or internal optimizations.

Sometimes my projects bump the revision number despite including several new commits into the new release and new features like a totally revamped interactive interface, like using sline. This was precisely the case from scalc 0.3.1 to 0.3.2 back in the day… And it will be very probably the case for cras, as all changes in the master branch so far are API backwards compatible with the latest 2.0.2 release.

Yeah, from my perspective as the lead developer, there’s been a lot going on, but for the user? OK, yes, a new interface that makes it easier to use these programs, but that doesn’t change the way these programs interact with other programs. And not to be an elitist, but this is what really matters from a versioning standpoint… because it’s all about signalling API users whether they’ll need to adapt their code or not. Changing a GUI or an interactive interface doesn’t ever come with the requirement to change how other components work with yours… It might just require someone to change their know-how.

That is, if you use versioning as meant to. If you use versioning as a marketing tool, we’re speaking different languages here. Microsoft’s infamous “versioning schemes” for Windows are the best example of this… but we all know that behind the brands Windows 95, 98, and Me, there were the actual version numbers Windows 4.00, 4.10, and 4.90, respectively… coming from Windows 3.11. That Windows XP was, logically, Windows NT 5.1/5.2… as it reunited both the NT (coming from Windows NT 3.51, NT 4.0, Win2k being NT 5.0…) and the MS-DOS-based lines…

Curiously, the Linux kernel is also another project that uses versioning in a very liberal way… at least for the major and minor numbers. This is possible mainly because backwards incompatible changes in Linux are extremely rare… I mean, Linux is a Unix clone… so… its APIs and ABIs have been fixed for ages. In a case like this, hey, let Linus change numbers as he wishes, frankly. I remember that the change from Linux 4.20 to 5.0 was so minuscule that the project didn’t even bother to change the version’s codename (both were Shy Crocodile), and… this was Linus’s announcement on the LKML back in the day:

But I’d like to point out (yet again) that we don’t do feature-based releases, and that “5.0” doesn’t mean anything more than that the 4.x numbers started getting big enough that I ran out of fingers and toes.

So Torvalds basically does whatever he likes, which is what we love him for, isn’t it?

So, Ariadna, You Actually Wanted to Talk about Version Numbers?

Yes :P Well, not exclusively. But I do feel this is important for me. I like software development and deployment to be transparent and I try to follow that as much as I can on my own projects, even if they are pretty irrelevant on the FOSS landscape.

Code is code, but there’s a lot of things that are supporting the whole structure of your project. The most obvious one might be documentation, as in manuals, user guides, etc., but knowing what your API is and document it via a version number is important. Of course it’s important for libraries, but equally for executables, regardless of the technology used to write them. I mean, I know I usually talk about C here, but think of Python code, which can easily be used both as executables and importable modules without any further modification.3 Not sure about Go or Rust, but I think they work like Python in this.4 In any case, if I’m using that code, I wanna know if there have been any changes I should be aware of.

And I myself like to be a good citizen and inform people who might be using my code that there have been changes to look out for when I’m releasing a new version. Of course, whatever happens between releases is pure anarchy and you definitely should be relying on any of my master branches at all unless you want to send in a patch or you really know what you’re doing. But that’s expected, isn’t it?

Personally, I do recommend people follow some kind of SemVer on their projects. This is a bit like code style guides: everyone ends up creating a personal version of a well-known guide. There’s room for some personal choice in all of this… but, again, documentation is the key to healthy FOSS’ing: if you state what you’re doing, people won’t be sending you angry emails… Or well, that’s what I’m hoping for!5

  1. ↩︎

  2. 0.Y numbers work in a different way, though… but those are reserved for development versions. ↩︎

  3. If the if __name__ == "__main__" idiom is used, that is. ↩︎

  4. Thanks to Alexey Yerin for pointing me out that this not the case in any of those two languages. ↩︎

  5. I haven’t gotten any to this date, fortunately! ↩︎