Source files and build scripts for my personal website
git clone
Download | Log | Files | Refs | README (14773B)

      1 # The man page reading club: sh(1) - part 1: shell grammar
      3 *This post is part of a [series](../../series)*
      5 After [last time's short entry](../2022-07-07-shutdown) and a
      6 relatively long hiatus, we are back in business with a big one!
      8 ## A new day
     10 *After a good night of sleep and a cup of whatever people call
     11 coffee in the post-apocalypse, you turn your computer back on. You
     12 would like to learn more stuff, but you are unsure where to start
     13 from. You vaguely remember a `man afterboot` being mentioned
     14 somewhere, so you start from there.*
     16 ```
     18    Starting out
     19      This document attempts to list items for the system administrator to
     20      check and set up after the installation and first complete boot of the
     21      system.  The idea is to create a list of items that can be checked off so
     22      that you have a warm fuzzy feeling that something obvious has not been
     23      missed.  A basic knowledge of UNIX is assumed, otherwise type:
     25 	   $ help
     26 ```
     28 *You do have some knowledge of UNIX, someone might call it "basic",
     29 but you believe "scattered" is a more appropriate adjective. In any
     30 case, a review won't hurt. You type the command*
     32 ```
     33 $ help
     34 ```
     36 *And a manual page shows up. You could have typed `man help` instead
     37 to get the same result. After skimming throught the introduction,
     38 you discover something worth digging into.*
     40 ```
     41    The Unix shell
     42      After logging in, some system messages are typically displayed, and then
     43      the user is able to enter commands to be processed by the shell program.
     44      The shell is a command-line interpreter that reads user input (normally
     45      from a terminal) and executes commands.  There are many different shells
     46      available; OpenBSD ships with csh(1), ksh(1), and sh(1).  Each user's
     47      shell is indicated by the last field of their corresponding entry in the
     48      system password file (/etc/passwd).
     49 ```
     51 *You have a look at `/etc/passwd` and you see that your user's shell
     52 is `ksh`. So you type `man ksh` and start reading.*
     54 ```
     56      ksh is a command interpreter intended for both interactive and shell
     57      script use.  Its command language is a superset of the sh(1) shell
     58      language.
     59 ```
     61 *You are quite rusty on the Math jargon - some of your friends used
     62 to talk like that in real life, but you never bothered to learn -
     63 but "superset" sounds like "it is larger than". Is this another
     64 [`less` vs `more`](../2022-06-08-more) kind of thing, where one
     65 command is just a simpler version of the other? Let's see what
     66 `sh(1)` has to say about it*
     68 ```
     69      This version of sh is actually ksh in disguise.
     70 ```
     72 *Ah-ah! Exactly as you thought. Just like the other time, you prefer
     73 to go with the simpler version. Enough of this "fun is precious"
     74 bullshit, you want to learn as soon as possible!*
     76 ## sh(1)
     78 *Follow along at [](*
     80 Despite having less features than more complex shells like `ksh`
     81 or `bash`, the manual page for `sh` is still very long. So we are
     82 going to split it into two or more parts.
     84 The main sections I intend to cover are BUILTINS, SHELL GRAMMAR and
     86 and explained in other sections, so I am probably going to skip
     87 these too.  I think we can skip the invocation options, since we
     88 are mostly going to run our shell implicitly when logging in or
     89 when executing a script. Finally, COMMAND HISTORY AND COMMAND LINE
     90 EDITING is best explained after we cover `vi(1)`, so we'll skip
     91 that too. This still leaves with a big chunk of the man page to
     92 discuss.
     94 A technical manual page is not a novel: the content is often laid
     95 out in an arbitrary order, to make it easier to find what you are
     96 looking for (e.g. in alphabetic order) and not to make a top-to-bottom
     97 read entertaining. So I felt like reordering things a bit: not only
     98 I will cover the sections in a differ order than what you find in
     99 the manual page, but I will also shuffle the content of each section
    100 when it make sense to me.
    102 Since I am very much a theoretical, grammar-first kind of person,
    103 my totally subjective best way to dive into this is starting with
    104 the grammar section!
    106 ## Part 1: shell grammar
    108 After reading the input, either from a file or from the standard
    109 input, `sh` does the following:
    111 1. It breaks the input into words and operators (special characters).
    112 2. It expands the text according to the rules in **Expansion** section below.
    113 3. It splits the text into commands and arguments.
    114 4. It performs input / output redirection (see the **Redirection** section below).
    115 5. It runs the commands.
    116 6. It waits for the commands to complete and collects the exit status.
    118 The next three sub-sections (Redirection, Expansion and Quoting) are found
    119 in the exact opposite order in the manual page.
    121 ### Redirection
    123 Together with *piping*, which we will cover in one of the next episodes,
    124 redirection is one of the key features of UNIX.
    126 ```
    127 	Redirection is used to open, close, or otherwise manipulate files, using
    128 	redirection operators in combination with numerical file descriptors.  A
    129 	minimum of ten (0-9) descriptors are supported; by convention standard
    130 	input is file descriptor 0, standard output file descriptor 1, and
    131 	standard error file descriptor 2.
    132 ```
    134 If the number `[n]` is not specified, it defaults to either `0`
    135 (standard input) or `1` (standard output) depending if the angled
    136 brackets are pointing to the left or to the right.
    138 The main redirectors are `[n]<file`, to read input from `file`
    139 instead of typing it in manually, and its counterpart `[n]>file`
    140 to write standard output (or whatever is described by the file
    141 descriptor `[n]`) to file. For example, if you want to log every
    142 error message of `command` to `file.log`, you can use
    144 ```
    145 $ command 2>file.log
    146 ```
    148 The `[n]>>file` redirector is similar, but it appends stuff to
    149 `file` instead of overwriting it. Both `>` and `>>` create the file
    150 if it does not exist.
    152 There is also `[n]<<`:
    154 ```
    155 [n]<<  This form of redirection, called a here document, is used to copy
    156        a block of lines to a temporary file until a line matching
    157        delimiter is read. When the command is executed, standard input
    158        is redirected from the temporary file to file descriptor n, or
    159        standard input by default.
    160 ```
    162 For example
    164 ```
    165 $ cat <<BYEBYE
    166 > one line,
    167 > another line
    168 > and so on
    169 > BYEBYE
    170 ```
    172 Outputs those three lines. It is useful in shell scripts, when you
    173 want to output a block of text. The variant `[n]<<-` strips out
    174 `Tab` characters.
    176 Another useful one is `[n]>&fd`, which "merges" the file descriptors
    177 `[n]` and `fd`. For example, if you want to make your command
    178 completely silent, you can merge standard output and standard error
    179 and redirect them both to `/dev/null` with
    181 ```
    182 $ command >&2 >/dev/null
    183 ```
    185 ### Expansion
    187 There are essentially five kinds of expansion that the shell performs:
    188 tilde expansion, parameter expansion, command expansion, arithmetic
    189 expansion and filename expansion.
    191 **Tilde expansion** is quite straightforward, so let's just quote
    192 the man page:
    194 ```
    195      Firstly, tilde expansion occurs on words beginning with the `~'
    196      character.	 Any characters following the tilde, up to the next colon,
    197      slash, or blank, are taken as a login name and substituted with that
    198      user's home directory, as defined in passwd(5).  A tilde by itself is
    199      expanded to the contents of the variable HOME.  This notation can be used
    200      in variable assignments, in the assignment half, immediately after the
    201      equals sign or a colon, up to the next slash or colon, if any.
    203 	   PATH=~alice:~bob/jobs
    204 ```
    206 **Parameters** can be variable names or special parameters. Variables
    207 can be assigned with the simple syntax `variable=value` and their
    208 value can be "accessed" with `$variable`. In case of ambiguity you
    209 need to enclose the variable name in curly braces `{}`: say you
    210 want to type the string `subroutines` and you have a variable
    211 `prefix=sub`. The shell will complain at a `$prefixroutines` about
    212 there being no variable with such name, so you have to use
    213 `${prefix}routines`.
    215 The most useful special parameters are:
    217 * Numbers `1`, `2`, `3`... that refer to the *positional parameters*:
    219 ```
    220     These parameters are set when a shell, shell script, or shell function is
    221     invoked.  Each argument passed to a shell or shell script is assigned a
    222     positional parameter, starting at 1, and assigned sequentially.
    223 ```
    225 * The number `0`, which refers to the name of the shell or of the shell
    226   script being executed.
    227 * The symbols `@` and `*` which expand to all positional parameters
    228   at once; they behave differently when enclosed in double quotes:
    229   with `"$@"` the parameters are split into fields, with `"$*"` they are not.
    231 There are some useful constructs to expand a parameter in special
    232 ways.  The constructs `${parameter:-[word]}` and `${parameter:=[word]}`
    233 expand to `[word]` if `parameter` is unset or empty, with the second
    234 one also assigning the value `[word]` to `parameter` for subsequent
    235 use. Instead, `${parameter:+[word]}` expands to `[word]` *unless*
    236 `parameter` is unset or empty, in which case it expands to the empty
    237 string. In all these cases, if the colon is omitted `[word]` is
    238 substituted only if `parameter` is unset (not if it is empty).
    240 Another useful one is `${#parameter}`, which expands to the length
    241 of `parameter`. Finally there are some constructs that can be used
    242 to remove prefixes or suffixes from the expansion of a parameter:
    244 | Construct | Effect |
    245 |:---:|:---:|
    246 | `${parameter%[word]}` | Delete smallest possible suffix matching word |
    247 | `${parameter%%[word]}` | Delete largest possible suffix matching word |
    248 | `${parameter#[word]}` | Delete smallest possible prefix matching word |
    249 | `${parameter##[word]}` | Delete largest possible prefix matching word |
    251 What unfortunately is not explained in the man page of `sh(1)` (but
    252 can be found in that of `ksh(1)`) is that `[word]` in this case can
    253 be a *pattern*.  See [glob(7)](
    254 for a description of patterns, which are the same that are used for
    255 filename expansion (with the exception that slashes and dots are
    256 treated as normal characters).
    258 For example, using `*` which means "any sequence of zero or more
    259 characters":
    261 ```
    262 $ x="we can,separate,stuff,with commas"
    263 $ echo ${x#*,}
    264 separate,stuff,with commas
    265 $ echo ${x##*,}
    266 with commas
    267 ```
    269 Then there is **command expansion**:
    271 ```
    272      Command expansion has a command executed in a subshell and the results
    273      output in its place.  The basic format is:
    275 	   $(command)
    276      or
    277 	   `command`
    279      The results are subject to field splitting and pathname expansion; no
    280      other form of expansion happens.  If command is contained within double
    281      quotes, field splitting does not happen either. 
    282 ```
    284 **Arithmetic expansion** uses the syntax `$((expression))`. An
    285 `expression` can be a combination of integers (no floating point
    286 arithmetic in the shell!), parameter names and the usual arithmetic
    287 operations. I won't copy them here; if you are familiar with C or
    288 C-like languages, you can use pretty much all the operations you
    289 are used to, including logic operations (resulting in 0 or 1),
    290 assignment operations like `+=` and bitwise operations like `~`,
    291 `&` and `<<`.  Even the *ternary if* `expression ? expr1 : expr2`
    292 is available.
    294 Finally, **filename expansion** uses the aforementioned rules of
    295 [glob(7)]( to expand
    296 filenames.  To sum them up:
    298 * As we have already seen, `*` expands to any sequence of characters.
    299 * `?` matches any single character.
    300 * `[..]` matches any character in place of the double dot, or any
    301   character *not* listed if the first is an exclamation mark.
    302 * `[[:class:]]` matches any character of a certain class; for example
    303   `class` could be `alnum` for alphanumeric characters or `upper` for
    304   uppercase letters.
    305 * `[x-y]` matches any character in the range between `x` and `y`.
    307 To illustrate what all of this means, check this out (the command `ls` is
    308 used to list all files in the current directory):
    310 ```
    311 $ ls
    312 box                  file3                mbox                 typescript
    313        file4                mnt                  videos
    314 file1                git                  music
    315 file2                mail                 phone-laptop-swap
    316 $ echo m*
    317 mail mbox mnt music
    318 $ echo m???
    319 mail mbox
    320 $ echo file[2-4]
    321 file2 file3 file4
    322 ```
    324 ### Quoting
    326 Sometimes we may want to write some of the special characters
    327 described above, such as dollar signs, without their special meaning.
    328 You can do so by *escaping*, or *quoting* them. There are essentially
    329 three ways to quote a character or a group of characters:
    331 * Backslash:
    333 ```
    334      A backslash (\) can be used to quote any character except a newline.  If
    335      a newline follows a backslash the shell removes them both, effectively
    336      making the following line part of the current one.
    337 ```
    339 This means that a backslash can also effectively be used to split
    340 long lines into multiple lines, for example for ease of editing a
    341 shell script.
    343 * Single quotes:
    345 ```
    346      A group of characters can be enclosed within single quotes (') to quote
    347      every character within the quotes.
    348 ```
    350 * And double quotes:
    352 ```
    353      A group of characters can be enclosed within double quotes (") to quote
    354      every character within the quotes except a backquote (`) or a dollar sign
    355      ($), both of which retain their special meaning.  A backslash (\) within
    356      double quotes retains its special meaning, but only when followed by a
    357      backquote, dollar sign, double quote, newline, or another backslash.  An
    358      at sign (@) within double quotes has a special meaning (see SPECIAL
    359      PARAMETERS, below).
    360 ```
    362 Basically the difference between single and double quotes is that
    363 the former turn literally everything they enclose into simple text,
    364 while the latter still parse and expand some special characters
    365 (for example the dollar sign `$` for variables).
    367 As an addition, remember that anything enclosed in single or double
    368 quotes is considered a single field (word). This was briefly mentioned
    369 in the Expansion section, but I skipped it. To illustrate what I
    370 mean, let's write a short script and run it first with some words
    371 as arguments and then with the same words enclosed in quotes:
    373 ```
    374 $ echo 'echo $#' >
    375 $ how many words are there
    376 5
    377 $ "how many words are there"
    378 1
    379 ```
    381 ## Until next time
    383 This was a very long post, but it made sense to keep all the grammar
    384 rules together. To finish this manual page we are going to need
    385 another long post, or two shorter ones.
    387 See you next time!
    389 *Next in the series: [sh(1) - part 2: commands and builtins](../2022-09-20-sh-2)*