Monday, 10 September 2012

out: an output utility


Whilst writing the tests for the Job Logging feature in Upstart, I discovered that I needed some very specific job behaviour that was not easy to produce using any simple existing utility I could find.

The Problem

What I wanted was a utility that would produce specified amounts of data to either standard output, standard error or direct to the terminal without using shell redirection. The reason for eschewing the shell being that Upstart is clever and intelligently determines whether a shell should be involved. From a user perspective, that's fantastic, but for testing purposes, I needed to force Upstart down particular code paths where it would not for example automatically pass the job through a shell. In fact the constraint was even more restrictive; what I really wanted to do was this:
Produce output (including null bytes) to stdout, stderr or the terminal without using any shell meta characters.
Unless I used some extremely esoteric shell, that effectively meant "no shells" ;-) ksh and zsh actually came close as their printf/print shell built-ins allow output to a specified file descriptor, but not to the terminal. Anyway, by this stage, I'd decided I wanted a generic utility, not some specific shell feature.

The Pragmatic Solution

For the Upstart tests, I ended up using various different command-line utilities for different tests that could produce the data I wanted. However, I couldn't help thinking there should be a better way. Yes, I could have written a bespoke C program or Python/Perl script to achieve my goal, but in the spirit of Larry Walls view that "Easy things should be easy", I thought that there should be a simpler method.

The Improved Solution

In the end this niggling problem got the better of me so I decided to write a tool to satisfy the above constraint. What I came up with is a small utility which sits somewhere between echo(1) and printf(1) (with just a dash of seq(1) thrown in) which, since it produces output, I've called simply "out".

Here's an example of how you could print "hello" direct to the terminal (the redirection here is just to show that you still get output even though we've discarded stdout and stderr output):
$ out -t hello >/dev/null 2>&1

This command below will echo "hello" to stdout, the terminal and stderr:
$ out hello -t -r 1 -e -r 1
Along with allowing output redirection to any file descriptor or the terminal without the support of shell redirection, it supports the standard C-style escape sequences (like '\n' and '\a'). It also allows you to redefine the escape character.

Additionally, out supports printf(1)-style Unicode escape sequences such as '\uFFFFFFFF' . After adding the Unicode support, I thought it would be rather fun to add ranges (similar to bash "sequence expressions") so out also supports some new sequences that allow sets of characters to be generated. These are great fun for exploring Unicode. Here's an example of generating the alphabet in lower-case:
$ out '\{a..z}\n'
How about some Greek?
$ out "\{α..ω}\n"

Or we could specify this using Unicode characters to get the same result:

$ out "\{\u03b1..\u03c9}\n"
And here's how you could generate the Unicode braille block:
$ out '\{\u2800..\u28FF}\n'
out also allows arbitrary delays to be introduced, repeats, basic random character generation and a few other tidbits. Here is the man page including a lot of examples:

OUT(1)         OUT(1)

       out - manual page for out

       out [OPTION]... [STRING]...

       Echo strings to specified output stream(s).

       -a, --intra-char=<char>
       Insert  specified  character  (which may be a 1-character escape
       character) between all output characters.

       -b, --intra-pause=<delay>
       Pause between writing each character.

       -e, --stderr
       Write subsequent strings to standard error (file descriptor 2).

       -h, --help
       This help text.

       -i, --interpret
       Interpret escape characters (default).

       -l, --literal
       Write literal strings only (disable escape characters).

       -o, --stdout
       Write subsequent strings to standard output (file descriptor 1).

       -p, --prefix=<prefix>
       Use <prefix> as escape prefix (default='\').

       -r, --repeat=<repeat>
       Repeat previous value <repeat> times.

       -s, --sleep=<delay>
       Sleep for <delay> amount of time.

       -t, --terminal
       Write subsequent strings directly to terminal.

       -u, --file-descriptor=<fd> Write to specified file descriptor.

       -x, --exit=<num>
       Exit with value <num>.

       out recognises C-style escape sequences as  used  by  printf(1) .   By
       default an  escape  sequence  is introduced by the backslash character
       ('\'), however this may be changed with the -p option.  out  also  sup‐
       ports some additional sequences:

       \0     - nul byte (hex value 0x00)

       \a     - alert (bell)

       \b     - backspace

       \c     - no further output

       \e     - escape character (used for changing terminal attributes)

       \f     - form feed

       \g     - generate pseudo-random printable character

       \n     - newline

       \oNNN  - byte with octal value NNN (1 to 3 digits)

       \r     - carriage return

       \t     - horizontal tab

       \uNNNN -  2  byte Unicode (ISO/IEC 10646) character with hex value NNNN
       (4 digits)

       - 4 byte Unicode  (ISO/IEC  10646)  character  with  hex  value
       NNNNNNNN (8 digits)

       \v     - vertical tab

       \xNN   - byte with hexadecimal value NN (1 to 2 digits)

       out also supports range escapes which allow a range of characters to be
       specified in a compact format.

       - specify a range by two 1-byte literal characters.

       - specify a range by two 3-byte octal values.

       - specify a range by two 2-byte Unicode values.

       - specify a range by two 4-byte Unicode values.

       - specify a range by two 2-byte hex values.

       Note that ranges take two values of the same type and the maximum width
       for that type must be specified.

       ·   Arguments are processed in order.

       ·   With the exception of '-x', arguments may be repeated any number of

       ·   All output will be sent to standard output until  an  output  redi‐
    rection  option is specified that changes the output stream (namely
    -e or -t (or their  long-option  equivalents),  or  if  output  has
    already been redirected -o (or its long-option equivalent)).

       ·   If  <str>  is  the empty string ("" or '') it will be treated as \0
    such that a nul byte will be displayed.

       ·   To cancel the effect of -a, specify a null string: -a ''.

       ·   If <repeat> is '-1', repeat forever.

       ·   Replace the 'Z' in the range formats  above with  the  appropriate

       ·   Ranges can be either ascending or descending.

       ·   <delay>  can  take  the  following  forms where <num> is a positive

    <num>ns : nano-seconds (1/1,000,000,000 second)
    <num>us : micro-seconds (1/1,000,000 second)
    <num>ms : milli-seconds (1/1,000 second)
    <num>cs : centi-seconds (1/100 second)
    <num>ds : deci-seconds (1/10 second)
    <num>s  : seconds
    <num>m  : minutes
    <num>h  : hours
    <num>h  : days
    <num>   : seconds

    If <num> is -1, wait until any signal is received.
    If signal is SIGNUM continue, else exit immediately.

       ·   Generated printable random characters may not  display  unless  you
    are using an appropriate font.

 # Print "foofoofoo" to stderr, followed by "barbar" to stdout.
 out "foo" -r 2 -o "bar" -r 1

 # Write 50 nul bytes direct to the terminal.
 out -t "" -r 49

 # Write continuous stream of nul bytes direct to the terminal,
 # 1 per second.
 out -b 1s -t '' -r -1

 # Display a greeting slowly (as a human might type)
 out -b 20cs "Hello, $USER.\n"

 # Display a "spinner" that loops 4 times.
 out -b 20cs -p % "%r|%r/%r-%r\%r" -r 3

 # Display all digits between zero and nine with a trailing
 # newline.
 out "\{0..9}\n"

 # Display slowly the lower-case letters of the alphabet,
 # backwards without a newline.
 out -b 1ds "\{z..a}"

 # Display upper-case 'ABC' with newline.
 out '\u0041\u0042\u0043\n'

 # Display 'foo' with newline.
 out '\o146\u006f\x6F\n'

 # Clear the screen.
 out '\n' -r $LINES

 # Write hello to stdout, stderr and the terminal.
 out 'hello' -t -r 1 -e -r 1

 # Display upper-case letters of the alphabet using octal
 # notation, plus a newline.
 out "\{\o101..\o132}"

 # Display 'h.e.l.l.o.' followed by a newline.
 out -a . "hello" -a '' "\n"

 # Display upper-case and lower-case letters of the alphabet
 # including the characters in-between, with a trailing newline.
 out "\{A..z}\n"

 # Display lower-case alphabet followed by reversed lower-case alphabet
 # with the digits zero to nine, then nine to zero on the next line.
 out "\{a..z}\{z..a}\n\{0..9}\{9..0}\n"

 # Display lower-case Greek letters of the alphabet.
 out "\{α..ω}"

 # Display cyrillic characters.
 out "\{Ѐ..ӿ}"

 # Display all printable ASCII characters using hex range:
 out "\{\x21..\x7e}"

 # Display all printable ASCII characters using 2-byte UTF-8 range:
 out "\{\u0021..\u007e}"

 # Display all printable ASCII characters using 4-byte UTF-8 range:
 out "\{\U00000021..\U0000007e}"

 # Display all braille characters.
 out "\{\u2800..\u28FF}"

 # Display 'WARNING' in white on red background.
 out '\e[37;41mWARNING\e[0m\n'

 # Generate 10 random characters.
 out '\g' -r 9

       Written by James Hunt <>

       Copyright © 2012 James Hunt <>

       GNU GPL version 3 or later <>.
       This  is  free  software:  you  are free to change and redistribute it.
       There is NO WARRANTY, to the extent permitted by law.

       echo(1) printf(1)

User Commands     2012-09-10    OUT(1)


The TODO file in the distribution contains a number of ideas to enhance outs abilities.


See Also

Tuesday, 4 September 2012

Call for Testing: Upstart stateful re-exec

If you'd like to help out testing the new "stateful re-exec" feature of Upstart which we hope to land for Ubuntu Quantal, see the details here:

In brief (more posts will follow), "stateful re-exec" allows Upstart to be restarted (or upgraded!) without rebooting your system. A basic restart or re-exec isn't difficult, but what stateful re-exec does is maintain Upstarts internal state over the re-exec such that it continues to track jobs and log job output.

This is an early cut of the code so don't install it on any critical systems. Additionally, there are some caveats...

... but we plan for that list to dwindle quickly over the coming days and weeks.