# The man page reading club: sh(1) - part 1: shell grammar *This post is part of a [series](../../series)* After [last time's short entry](../2022-07-07-shutdown) and a relatively long hiatus, we are back in business with a big one! ## A new day *After a good night of sleep and a cup of whatever people call coffee in the post-apocalypse, you turn your computer back on. You would like to learn more stuff, but you are unsure where to start from. You vaguely remember a `man afterboot` being mentioned somewhere, so you start from there.* ``` DESCRIPTION Starting out This document attempts to list items for the system administrator to check and set up after the installation and first complete boot of the system. The idea is to create a list of items that can be checked off so that you have a warm fuzzy feeling that something obvious has not been missed. A basic knowledge of UNIX is assumed, otherwise type: $ help ``` *You do have some knowledge of UNIX, someone might call it "basic", but you believe "scattered" is a more appropriate adjective. In any case, a review won't hurt. You type the command* ``` $ help ``` *And a manual page shows up. You could have typed `man help` instead to get the same result. After skimming throught the introduction, you discover something worth digging into.* ``` The Unix shell After logging in, some system messages are typically displayed, and then the user is able to enter commands to be processed by the shell program. The shell is a command-line interpreter that reads user input (normally from a terminal) and executes commands. There are many different shells available; OpenBSD ships with csh(1), ksh(1), and sh(1). Each user's shell is indicated by the last field of their corresponding entry in the system password file (/etc/passwd). ``` *You have a look at `/etc/passwd` and you see that your user's shell is `ksh`. So you type `man ksh` and start reading.* ``` DESCRIPTION ksh is a command interpreter intended for both interactive and shell script use. Its command language is a superset of the sh(1) shell language. ``` *You are quite rusty on the Math jargon - some of your friends used to talk like that in real life, but you never bothered to learn - but "superset" sounds like "it is larger than". Is this another [`less` vs `more`](../2022-06-08-more) kind of thing, where one command is just a simpler version of the other? Let's see what `sh(1)` has to say about it* ``` This version of sh is actually ksh in disguise. ``` *Ah-ah! Exactly as you thought. Just like the other time, you prefer to go with the simpler version. Enough of this "fun is precious" bullshit, you want to learn as soon as possible!* ## sh(1) *Follow along at [man.openbsd.org](https://man.openbsd.org/OpenBSD-7.1/sh)* Despite having less features than more complex shells like `ksh` or `bash`, the manual page for `sh` is still very long. So we are going to split it into two or more parts. The main sections I intend to cover are BUILTINS, SHELL GRAMMAR and COMMANDS. Parts of SPECIAL PARAMETERS and ENVIRONMENT are quoted and explained in other sections, so I am probably going to skip these too. I think we can skip the invocation options, since we are mostly going to run our shell implicitly when logging in or when executing a script. Finally, COMMAND HISTORY AND COMMAND LINE EDITING is best explained after we cover `vi(1)`, so we'll skip that too. This still leaves with a big chunk of the man page to discuss. A technical manual page is not a novel: the content is often laid out in an arbitrary order, to make it easier to find what you are looking for (e.g. in alphabetic order) and not to make a top-to-bottom read entertaining. So I felt like reordering things a bit: not only I will cover the sections in a differ order than what you find in the manual page, but I will also shuffle the content of each section when it make sense to me. Since I am very much a theoretical, grammar-first kind of person, my totally subjective best way to dive into this is starting with the grammar section! ## Part 1: shell grammar After reading the input, either from a file or from the standard input, `sh` does the following: 1. It breaks the input into words and operators (special characters). 2. It expands the text according to the rules in **Expansion** section below. 3. It splits the text into commands and arguments. 4. It performs input / output redirection (see the **Redirection** section below). 5. It runs the commands. 6. It waits for the commands to complete and collects the exit status. The next three sub-sections (Redirection, Expansion and Quoting) are found in the exact opposite order in the manual page. ### Redirection Together with *piping*, which we will cover in one of the next episodes, redirection is one of the key features of UNIX. ``` Redirection is used to open, close, or otherwise manipulate files, using redirection operators in combination with numerical file descriptors. A minimum of ten (0-9) descriptors are supported; by convention standard input is file descriptor 0, standard output file descriptor 1, and standard error file descriptor 2. ``` If the number `[n]` is not specified, it defaults to either `0` (standard input) or `1` (standard output) depending if the angled brackets are pointing to the left or to the right. The main redirectors are `[n]file` to write standard output (or whatever is described by the file descriptor `[n]`) to file. For example, if you want to log every error message of `command` to `file.log`, you can use ``` $ command 2>file.log ``` The `[n]>>file` redirector is similar, but it appends stuff to `file` instead of overwriting it. Both `>` and `>>` create the file if it does not exist. There is also `[n]<<`: ``` [n]<< This form of redirection, called a here document, is used to copy a block of lines to a temporary file until a line matching delimiter is read. When the command is executed, standard input is redirected from the temporary file to file descriptor n, or standard input by default. ``` For example ``` $ cat < one line, > another line > and so on > BYEBYE ``` Outputs those three lines. It is useful in shell scripts, when you want to output a block of text. The variant `[n]<<-` strips out `Tab` characters. Another useful one is `[n]>&fd`, which "merges" the file descriptors `[n]` and `fd`. For example, if you want to make your command completely silent, you can merge standard output and standard error and redirect them both to `/dev/null` with ``` $ command >/dev/null 2>&1 ``` ### Expansion There are essentially five kinds of expansion that the shell performs: tilde expansion, parameter expansion, command expansion, arithmetic expansion and filename expansion. **Tilde expansion** is quite straightforward, so let's just quote the man page: ``` Firstly, tilde expansion occurs on words beginning with the `~' character. Any characters following the tilde, up to the next colon, slash, or blank, are taken as a login name and substituted with that user's home directory, as defined in passwd(5). A tilde by itself is expanded to the contents of the variable HOME. This notation can be used in variable assignments, in the assignment half, immediately after the equals sign or a colon, up to the next slash or colon, if any. PATH=~alice:~bob/jobs ``` **Parameters** can be variable names or special parameters. Variables can be assigned with the simple syntax `variable=value` and their value can be "accessed" with `$variable`. In case of ambiguity you need to enclose the variable name in curly braces `{}`: say you want to type the string `subroutines` and you have a variable `prefix=sub`. The shell will complain at a `$prefixroutines` about there being no variable with such name, so you have to use `${prefix}routines`. The most useful special parameters are: * Numbers `1`, `2`, `3`... that refer to the *positional parameters*: ``` These parameters are set when a shell, shell script, or shell function is invoked. Each argument passed to a shell or shell script is assigned a positional parameter, starting at 1, and assigned sequentially. ``` * The number `0`, which refers to the name of the shell or of the shell script being executed. * The symbols `@` and `*` which expand to all positional parameters at once; they behave differently when enclosed in double quotes: with `"$@"` the parameters are split into fields, with `"$*"` they are not. There are some useful constructs to expand a parameter in special ways. The constructs `${parameter:-[word]}` and `${parameter:=[word]}` expand to `[word]` if `parameter` is unset or empty, with the second one also assigning the value `[word]` to `parameter` for subsequent use. Instead, `${parameter:+[word]}` expands to `[word]` *unless* `parameter` is unset or empty, in which case it expands to the empty string. In all these cases, if the colon is omitted `[word]` is substituted only if `parameter` is unset (not if it is empty). Another useful one is `${#parameter}`, which expands to the length of `parameter`. Finally there are some constructs that can be used to remove prefixes or suffixes from the expansion of a parameter: | Construct | Effect | |:---:|:---:| | `${parameter%[word]}` | Delete smallest possible suffix matching word | | `${parameter%%[word]}` | Delete largest possible suffix matching word | | `${parameter#[word]}` | Delete smallest possible prefix matching word | | `${parameter##[word]}` | Delete largest possible prefix matching word | What unfortunately is not explained in the man page of `sh(1)` (but can be found in that of `ksh(1)`) is that `[word]` in this case can be a *pattern*. See [glob(7)](https://man.openbsd.org/OpenBSD-7.1/glob.7) for a description of patterns, which are the same that are used for filename expansion (with the exception that slashes and dots are treated as normal characters). For example, using `*` which means "any sequence of zero or more characters": ``` $ x="we can,separate,stuff,with commas" $ echo ${x#*,} separate,stuff,with commas $ echo ${x##*,} with commas ``` Then there is **command expansion**: ``` Command expansion has a command executed in a subshell and the results output in its place. The basic format is: $(command) or `command` The results are subject to field splitting and pathname expansion; no other form of expansion happens. If command is contained within double quotes, field splitting does not happen either. ``` **Arithmetic expansion** uses the syntax `$((expression))`. An `expression` can be a combination of integers (no floating point arithmetic in the shell!), parameter names and the usual arithmetic operations. I won't copy them here; if you are familiar with C or C-like languages, you can use pretty much all the operations you are used to, including logic operations (resulting in 0 or 1), assignment operations like `+=` and bitwise operations like `~`, `&` and `<<`. Even the *ternary if* `expression ? expr1 : expr2` is available. Finally, **filename expansion** uses the aforementioned rules of [glob(7)](https://man.openbsd.org/OpenBSD-7.1/glob.7) to expand filenames. To sum them up: * As we have already seen, `*` expands to any sequence of characters. * `?` matches any single character. * `[..]` matches any character in place of the double dot, or any character *not* listed if the first is an exclamation mark. * `[[:class:]]` matches any character of a certain class; for example `class` could be `alnum` for alphanumeric characters or `upper` for uppercase letters. * `[x-y]` matches any character in the range between `x` and `y`. To illustrate what all of this means, check this out (the command `ls` is used to list all files in the current directory): ``` $ ls box file3 mbox typescript count_args.sh file4 mnt videos file1 git music file2 mail phone-laptop-swap $ echo m* mail mbox mnt music $ echo m??? mail mbox $ echo file[2-4] file2 file3 file4 ``` ### Quoting Sometimes we may want to write some of the special characters described above, such as dollar signs, without their special meaning. You can do so by *escaping*, or *quoting* them. There are essentially three ways to quote a character or a group of characters: * Backslash: ``` A backslash (\) can be used to quote any character except a newline. If a newline follows a backslash the shell removes them both, effectively making the following line part of the current one. ``` This means that a backslash can also effectively be used to split long lines into multiple lines, for example for ease of editing a shell script. * Single quotes: ``` A group of characters can be enclosed within single quotes (') to quote every character within the quotes. ``` * And double quotes: ``` A group of characters can be enclosed within double quotes (") to quote every character within the quotes except a backquote (`) or a dollar sign ($), both of which retain their special meaning. A backslash (\) within double quotes retains its special meaning, but only when followed by a backquote, dollar sign, double quote, newline, or another backslash. An at sign (@) within double quotes has a special meaning (see SPECIAL PARAMETERS, below). ``` Basically the difference between single and double quotes is that the former turn literally everything they enclose into simple text, while the latter still parse and expand some special characters (for example the dollar sign `$` for variables). As an addition, remember that anything enclosed in single or double quotes is considered a single field (word). This was briefly mentioned in the Expansion section, but I skipped it. To illustrate what I mean, let's write a short script and run it first with some words as arguments and then with the same words enclosed in quotes: ``` $ echo 'echo $#' > count_args.sh $ count_args.sh how many words are there 5 $ count_args.sh "how many words are there" 1 ``` ## Until next time This was a very long post, but it made sense to keep all the grammar rules together. To finish this manual page we are going to need another long post, or two shorter ones. See you next time! *Next in the series: [sh(1) - part 2: commands and builtins](../2022-09-20-sh-2)*