 spent some time today working on this diagram of how the ASCII control characters work in unix there are a lot of mistakes/missing nuance but I think it's really interesting how little structure there is. Special codes (like `3` for SIGINT) that are handled by the OS are mixed with just regular keypresses (like `13` for "enter”) which are mixed with codes that are handled by the application (like `1` for Ctrl-A in readline) (not looking for history lessons right now) 82 comments
some other things I think are interesting about the control codes: * some of the "OS terminal driver stuff" is also sometimes handled by readline  @b0rk several of the readline control codes are implemented as keyboard shortcuts in some common text editing widgets, particularly Ctrl+A/E @b0rk That is also another hint for the 'j' in vim: go down one line, just like ctrl+j does in a terminal. Or is this just a coincident? @bartavi @b0rk Likely coincidence. Have a look at "man ascii" and look at characters Decimal 1 to 26. Those are for the most part what you will get if it doesn't already do something special (like Ctrl+C). For example, Ctrl+H is character 8, which is backspace (but only the backspace character). Ctrl+I is a tab, and Ctrl+J is a newline '\n'. (This means Ctrl+M is actually a carriage return '\r', but your os likely translates both into '\n' in the input that your terminal programs receive). @b0rk 0x7f is usually called "delete" or "DEL", to constrast it with 0x08 which is "backspace", and they are different and sometimes the terminal gets confused. But your chart calls them both "backspace", which I think might tend to add to the amount of confusion in the universe. @mjd thanks that's a good point -- I'm very confused about what Ctrl-H does in practice -- to me it feels like 0x08 is just kind of some weird old cruft that has no real purpose but maybe that's not true (if it does have a purpose I'd be interested to know what it is! I don't have a backspace key on my keyboard right now but my impression is that the backspace key is supposed to map to an ASCII DEL)  @b0rk if you type "abcdef" at a prompt and then put your cursor between c and d (abc|def) then: 0x08 (backspace) usually produces ab|def 0x7f (delete) usually produces abc|ef @b0rk @mjd My memory is that there was a big schism in the physical terminal world about what the 'backspace' key generated. Many terminals had it generate Ctrl-H, but some used DEL; I believe DEC terminals such as the VT-100 were in the DEL camp, and they were very popular. This carried over to Unix workstations, with vendors doing different mappings (eg SGI made their 'backspace' key generate Ctrl-H in a default configuration; DEC used DEL of course). @b0rk @mjd Today, X on Linux has a 'BackSpace' keysym and a 'Delete' keysym, generated by their respective keyboard keys, but I believe that all terminal emulators map both of them to DEL by default (xterm and gnome-terminal certainly do). Things like browsers tend to make the big BackSpace key delete characters to the left of the cursor (what you expect) and the 'Delete' key delete to the right, which can be a surprise.  @b0rk @mjd I see a lot of people have posted about history, but as I see it the _effects_ of the history are to create two current schools of thought. 1. the Backspace key (above Return) is 0x08, and the Del key (on the nav keypad) is 0x7F 2. Backspace is 0x7F, and Del is some sequence like ESC[3~ (similar to the rest of that keypad) Emacs users like #2 because it frees up 0x08 to be ^H for help. But enough people like #1 that lots of software all needs a config option! @b0rk @mjd The Debian Policy Manual has an entire section on this: https://www.debian.org/doc/debian-policy/ch-opersys.html#keyboard-configuration @b0rk Fun trivia on the Ctrl-1 and Ctrl-M and so on thing: non-terminal graphical programs with direct access to keypress information can bind them (of course). This is especially fun for programs that can operate either in the terminal or as stand-alone graphical applications, like GNU Emacs, because you can innocently create a binding for yourself that only works in GUI mode (which you might not notice for a while if you only rarely use the editor in a terminal).  @cks @b0rk like how avy-mode¹ recommends binding Ctrl-: in its default docs², with the consequence that it's unusable by folks who use terminal emacs (or a combination of GUI and terminal emacs panes connecting to a shared emacs daemon). Unless they're wizardly enough to feel comfortable reading all documentation about it when an interpretation filter in their brain! ¹: https://karthinks.com/software/avy-can-do-anything/ ²: https://github.com/abo-abo/avy?tab=readme-ov-file#avy-goto-char  @b0rk web-based injections attacks make a bit more sense, given that this was part of the basis for stuff @b0rk I also use Ctrl-K (delete from cursor to right) a lot! I just reminded myself that Ctrl-U deletes from the cursor to the left. I tend to use Ctrl-W instead.   @b0rk the readline codes are theoretically mnemonic based on the letter, like ^R for reverse search, but a lot of them are a real stretch (^Y "yank" = paste???) @b0rk minor nitpick, ^D is the EOT (End-of-Transmission) control character, which the unix terminal driver translates to an EOF (which technically isn't a character) indicator/condition if it's at the start of a line  Did you notice that the control character is the same as the corresponding letter you type with a change in the high order octal place? Ex bell is octal 007 and G is octal 107. I find this really useful when I can't remember how to type a particular control character, I can just look at the ASCII man page to figure it out. @Moore interesting what's an example of a control character you might look at the ASCII man page to type in? I know about this correspondance but I've never found a use for it @b0rk anything other than ^C. I have only used it for practical reasons once or twice. @b0rk It is worthwhile to think of "Unix" and "readline" separately. "Unix" provides terminal driver functionality, and cooked mode is part of the driver. Originally, all line editing functionality was provided by the tty driver, and different Unix variants had different features. The BSD driver was way more feature rich than the Unix System V driver. One thing all Unix tty drivers have in common, however, is that they have programmable control characters. 1/ @hanshuebner I don't really understand what you mean, I think it's obvious that "readline" exists at a very different level than the terminal driver? what's interesting to me is that from a user’s perspective it's very unclear which things are being handled by the OS and which are passed through to the application and they're just kind of chaotically mixed together @b0rk I don't think that it is all that obvious that readline is not part of the driver. Providing kernel-level line editing basically was a necessity in the 16bit era, but when machines got more powerful, providing these things in user mode became more common. What that also meant is that there was no single line editing system, but each user-mode program could (and needed) to provide its own. GNU readline, albeit popular, was really hindered by its license. @b0rk With a user-mode line editor, it is only the control characters that generate interrupts which require driver support, and modern implementations don't even need to rely on the driver to send signals (i.e. they treat Ctrl-C like any other character and send a SIGINT if they want to). @b0rk The defaults for the control characters varied between systems. System V originally had @ as the kill character and DEL as interrupt, which was rather annoying when you used a terminal that had DEL where you would have the BS key. When machines got faster, it became feasible to have a user-mode line editing program do the input processing. tcsh was the first popular shell to have line editing with arrow key support. ksh provided vi-style editing in a hybrid setting. 2/ @b0rk I bore you with these historical details because I think that the question "what control characters does Unix use" cannot really be answered on that level in a meaningful manner. It may make sense to discuss common driver-based and user mode line editing facilities, but generally, in the Unix world, they are all configurable and the defaults changed over time. Other systems (i.e. VMS or DG/OS) were way less flexible, which meant that they required specific terminals. 3/3 @hanshuebner hmm it feels a lot more fixed to me than that, like I know there are a lot of unixes out there but personally as someone who has only ever used Mac and Linux machines and would absolutely never change my `stty` settings in a million years, for me there are a lot of assumptions it's very fair to make about how these things are going to work like it's hard to imagine that very many people are going to remap SIGINT to something other than Ctrl-C even if theoretically you can @b0rk I don't disagree. I guess for most people, the fact that a terminal driver exists and can be misconfigured is just a historical nuisance and not something that they can or need to take advantage of. @hanshuebner for me it's useful to know about the terminal driver because it means that when I’m using some terrible program without line editing support (which happens more often than I'd like!) I know that the terminal driver will still let me do `Ctrl-W` or `Ctrl-U` when I make a mistake @b0rk Absolutely! This is why I think that tty driver functionality and user-mode line editing behavior deserve separate treatment. I think it is still useful to know that the driver exists and when it is active. Also, it is worth knowing that BSD derived systems have the status character (Ctrl-T) and that incorrect setting of the erase character can bite you. @b0rk in a literal sense, ctrl-H *is* backspace. The ERASE function in termios has historically been set to either DEL or BACKSPACE, depending on the keyboard layout and configuration of the terminal being used. This is why many terminal emulators still have a configuration setting to indicate what the Backspace/DEL/<- key on your keyboard should generate. Whether user space code honours the termios setting when running in raw mode depends on the user space code... @wfk on my machine ctrl-H is not backspace, would be curious to see the output of stty -a on your machine to see how it works there! @b0rk we may be using different definitions of backspace here. I suspect you are talking about what is named the ERASE function in termios: undo the previously typed character on the line. I use backspace in the ASCII meaning, which is the ASCII code associated with ctrl-H. If your keyboard does not generate that ASCII code when you press ctrl-H, that's a seriously non-standard setup. This is a function of the terminal (emulator), not of the tty driver. @wfk yea I guess what I'm trying to figure out is whether 0x08 has much of a real purpose today or whether it's mostly cruft  @b0rk @hanshuebner And people who want to can reset tty or readline config, but they are controlled in different ways. (I disable emacs style editing everywhere I can use vi style.)  @hanshuebner talking about software maintenance, the context around it changes which changes what design and abstraction makes sense. Given today's terminal context, maybe one could design a new tty layer that is much simpler by making more assumptions about the capabilities of the terminals and apps.  @b0rk wait wait wait, sigint is signal 3 (so presumably kill -3 pid works?) and C is the third letter, and that’s why ctrl-C quits things?? How fascinating! I had no idea! Every time you do a thing I learn something :) @b0rk part of the lack of structure is due to the historical development of how these are interpreted. The kernel tty driver termios subsystem interprets some characters, depending on the termios ioctl settings. Some application level code, including, but not limited to, libreadline, changes these settings and then interprets some of these in user space. How libreadline interprets certain characters depends on the implementation and the settings (emacs vs vi mode and more). @b0rk so, as @hanshuebner points out, a lot depends on exactly what software you are currently running, and how you have configured the various parts of the system. Now, most users will to a significant extent just use whatever default settings their system provides, and most Linux distros have very similar defaults, so most of this will be mostly similar for most users most of the time. Which makes it all the more frustrating when they run into something that is just a bit different...   @b0rk Don't be too dogmatic. These days they're all changeable, and some of us change them. I'm fond of DEL as interrupt, not just because it's historic but because it's a simple single key to smash. Sticking to @ as my kill character is more obstreperous. I am no stick-in-the-mud, though: BS for erase since I started using Unix. I happen to know that ken used # for erase as late as the latter 1980s.  @b0rk under NetBSD (and maybe others, but sadly not Linux) ctrl+T sends a SIGINFO to the foreground process. Some long-running tools like dd will respond to this signal with a status line saying how many blocks transferred so far, and the default signal handler prints the PID and some CPU usage stats for the process! A really nice feature I miss a lot on Linux 😥  @jleedev haha I'm so curious about the history of that “Ctrl-?, which is wrong" statement (I have no idea why it would be wrong but obviously the person who wrote that felt strongly about it!)  @b0rk It seems mostly disused these days, but Ctrl-O used to be bound to DISCARD by default. When typed by the user, the terminal driver would discard (“flush”) all output to the terminal until the program next read from the terminal. This was useful if some command produced a lot of unwanted output, especially on a slow terminal. I first saw this on TOPS-20, and it also appeared in BSD Unix. @b0rk Another obscure character is Ctrl-Y (DSUSP, delayed suspend). Similar function to Ctrl-Z, which sends SIGTSTP immediately. However, Ctrl-Y is buffered up by the terminal driver with other input. When it is about to be read by the program, the driver sends SIGTSTP at that point and discards the Ctrl-Y. I think both DSUSP and DISCARD are BSDisms. Various man pages seem to indicate they aren’t supported on Linux.  @stuartmarks @b0rk I remember using Ctrl-O on vt220 terminals to shut them up. Not sure if it was a thing on paper roll tty-33 and Dec-10 because that was too long ago… @stuartmarks @b0rk omg where were you in 86 when I was trying to get stuff done using the godawful slow-ass terminals on our uni's PDP11?  @tomjennings @b0rk I had never considered this stuff until two minutes ago, and now I’m intrigued. Thank you 😊 @b0rk I love how the ITS operating system shows its (ha) influence on 36% of this table.  @b0rk This may be a silly question, but how do some of these key combinations work with different keyboard layouts? For example, on a Swedish (mac) layout, the @-symbol is alt-gr-2; Do you type ctrl-alt-gr-2 to get the null character? @Charlesflorian @b0rk pretty sure it’s just ctrl-2. On most layouts the ctrl-symbol codes are (also) mapped to ctrl-2...ctrl-8 |
Gregory's Server
@b0rk this absolutely would have saved me an afternoon of work when I ran into \x0b in a data stream nested inside a newline-delimited data stream!