aboutsummaryrefslogtreecommitdiff
path: root/src
diff options
context:
space:
mode:
authorSebastiano Tronto <sebastiano@tronto.net>2023-12-03 22:46:15 +0100
committerSebastiano Tronto <sebastiano@tronto.net>2023-12-03 22:46:15 +0100
commit8ecef139cf51b794a9a7e5a2686dbca3581340e1 (patch)
treef79ade7b9d44f9d42638c84c7a7492ca945f58ec /src
parentce25bbd863904ac7784b647b6b8e22196e7674ff (diff)
downloadsebastiano.tronto.net-8ecef139cf51b794a9a7e5a2686dbca3581340e1.tar.gz
sebastiano.tronto.net-8ecef139cf51b794a9a7e5a2686dbca3581340e1.zip
New post, fixed some stuff
Diffstat (limited to 'src')
-rw-r--r--src/blog/2023-06-16-regex/regex.md4
-rw-r--r--src/blog/2023-08-20-grep/grep.md2
-rw-r--r--src/blog/2023-12-03-sed/sed.md402
-rw-r--r--src/series/series.md11
4 files changed, 414 insertions, 5 deletions
diff --git a/src/blog/2023-06-16-regex/regex.md b/src/blog/2023-06-16-regex/regex.md
index e6c46f1..bd77b15 100644
--- a/src/blog/2023-06-16-regex/regex.md
+++ b/src/blog/2023-06-16-regex/regex.md
@@ -1,5 +1,7 @@
1# UNIX text filters, part 0 of 3: regular expressions 1# UNIX text filters, part 0 of 3: regular expressions
2 2
3*This post is part of a [series](../../series)*
4
3One of the most important features of UNIX and its descendants, if 5One of the most important features of UNIX and its descendants, if
4not *the* most important feature, is input / output redirection: 6not *the* most important feature, is input / output redirection:
5the output of a command can be displayed to the user, written to a 7the output of a command can be displayed to the user, written to a
@@ -199,3 +201,5 @@ But, quoting from the manual page:
199I hope you enjoyed this post, despite the lack of practical examples. 201I hope you enjoyed this post, despite the lack of practical examples.
200If you want to see more applications of regular expressions, stay 202If you want to see more applications of regular expressions, stay
201tuned for the next entries on grep, sed and awk! 203tuned for the next entries on grep, sed and awk!
204
205*Next in the series: [grep](../2023-08-20-grep)*
diff --git a/src/blog/2023-08-20-grep/grep.md b/src/blog/2023-08-20-grep/grep.md
index 47f414b..597d8f7 100644
--- a/src/blog/2023-08-20-grep/grep.md
+++ b/src/blog/2023-08-20-grep/grep.md
@@ -250,3 +250,5 @@ advanced tools, such as `sed` and `awk`: the "read one line, process
250it, print something" idea is common to all three of them. 250it, print something" idea is common to all three of them.
251 251
252Stay tuned for the part 2: `sed`! 252Stay tuned for the part 2: `sed`!
253
254*Next in the series: [sed](../2023-12-03-sed)*
diff --git a/src/blog/2023-12-03-sed/sed.md b/src/blog/2023-12-03-sed/sed.md
new file mode 100644
index 0000000..c422879
--- /dev/null
+++ b/src/blog/2023-12-03-sed/sed.md
@@ -0,0 +1,402 @@
1# UNIX text filters, part 2 of 3: sed
2
3*This post is part of a [series](../../series)*
4
5After the first (or second, depending on how you prefer to call ordinals
6in a 0-based system) episode on [`grep`](../2023-08-20-grep) we are ready
7to look at `sed`, the *stream* editor!
8
9You can think of `sed` as the weird cousin of [`ed`](../2022-12-24-ed),
10the standard editor, as they share much of their syntax. You could
11argue that `ed` is the weirder one, though.
12
13On the other hand, the *stream* part of `sed` is very peculiar,
14and I prefer to think about it as a sort of `grep` that can not
15only pick the desired lines, but also edit them. You can decide
16which point of view you prefer after reading this post!
17
18## Basic usage
19
20The way sed works is easy to summarize: text is read from standard input
21(or from a given file) line by line, a command is applied to each line,
22and the output is printed. Pretty much the same as for `grep`, except
23for the *a command is applied* part. Therefore, the power of `sed`
24comes from the available commands.
25
26A typical sed command is run like this:
27
28```
29$ sed [options] 'command' [file ...]
30```
31
32Instead of diving into the formal definition of the
33grammar of sed, or following the
34[manual page](https://man.openbsd.org/sed),
35let's start with the basics.
36
37### Replacing text: the `s` command
38
39Most of the times I use `sed`, and pretty much every time I use it
40in an interactive shell, I just use the *substitution command* `s`.
41If you have used `sed` in the past, chances are you have used `s`.
42
43As a basic example, say you want to replace all occurrences of the word
44"dog" with the word "cat". Then you can use `sed s/dog/cat/g`:
45
46```
47$ echo "I love dogs! My dog is cute" | sed 's/dog/cat/g'
48I love cats! My cat is cute
49```
50
51If you omit the `g` at the end, only the first occurrence on each line
52is replaced:
53
54```
55$ echo "I love dogs! My dog is cute
56> Another dog line" | sed 's/dog/cat/g'
57I love cats! My dog is cute
58Another cat line
59```
60
61### Regular expressions
62
63Plain text substitution works fine in educational examples, but it may
64fail in real-world use cases:
65
66```
67$ echo "Dogs are cool. My dog is called Doge." | sed 's/dog/cat/g'
68Dogs are cool. My cat is called Doge.
69```
70
71Luckily, regular expressions come to rescue! The first part of
72a substitution command can be a (basic) regular expression. Most
73versions of `sed` also support extended regular expressions via
74the `-E` or `-r` options, though this is not mandated by
75[POSIX](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sed.html).
76Check your local manual page, and see also the section **BSD sed vs GNU sed**
77below. For more info on regular expressions,
78see [part 0](../2023-06-16-regex) of this series.
79
80Back to our example. We can use:
81
82```
83$ echo "Dogs are cool. My dog is called Doge." | sed 's/[Dd]og/cat/g'
84cats are cool. My cat is called cate.
85```
86
87Ok, we had one problem and we solved it. Now we have two problems.
88
89One problem is that the name of the dog was also canged, as it contains the
90word "Dog". This can be fixed by using a more complicated regular
91expression that matches word boundaries. With GNU `sed` (the default
92in most Linux distros) the regular expression that matches dog or
93Dog only when it is a word is `\b[Dd]og\b`, while on most BSD systems
94it is `[[:<:]][Dd]og[[:>:]]`. As far as I know, none of these is
95mandated by POSIX; avoid them if you are writing portable shell
96scripts.
97
98The second problem is that the replacement text does not respect
99the replaced text's capitalization. One simple way to solve this
100is using multiple commands.
101
102### Multiple commands
103
104A `sed` command can be a composition of multiple commands. This is
105true not only for `s`, but also for all other commands that we have
106not seen yet.
107
108Commands are concatenated with a semi-colon. For example:
109
110```
111$ echo "Dogs are great, I love dogs!" | sed 's/dog/cat/g ; s/Dog/Cat/g'
112Cats are great, I love cats!
113```
114
115Concatenated commands are applied, in the order they appear, to
116every line. Beware that subsequent commands operate on the modified
117line! For example:
118
119```
120$ echo "dogs and cats" | sed 's/dog/cat/g ; s/cat/dog/g'
121dogs and dogs
122```
123
124There are other ways of giving `sed` multiple commands to execute
125for each line. Similarly to `grep`, you can use `-e COMMAND -e ...`
126to list more commands directly, or `-f FILE` to let sed read the
127commands from a file.
128
129### Little trick: change the separator to avoid escaping slashes
130
131For the `s` command, the slash `/` is a special character; if you
132want to use it in your regular expression or in your substitution
133text, you need to escape it with a backslash `\`. For example, to
134change all the slashes to backslashes you can use something like:
135
136```
137sed 's/\//\\/g'
138```
139
140But you don't have to use the slash as a separator - actually, you can
141use any character other than a backslash or a newline. If you use a
142different separator, you don't need to escape slashes - though you do
143need to escape whatever separator you choose instead. For example,
144to perform the same substitution as above you can use a pipe `|` as
145a separator:
146
147```
148sed 's|/|\\|g'
149```
150
151A bit better, but you still need to escape backslashes.
152
153### Addresses
154
155In general, `sed` commands have the following form:
156
157```
158[address[,address]]function[arguments]
159```
160
161Addresses specify the range of lines of the text on which the given
162function is applied. If no address is given, the command is applied
163to all lines. With only one address the command applies to that
164single line. Addresses can be also a dollar sign `$`, matching the
165last line, or a regular expression surrounded by slashes (e.g.
166`/re/`), matching all the lines that match the expression.
167
168Does this remind you of something? It should, if you have read my
169[post on `ed`, the standard editor](../2022-12-24-ed). Addresses in
170`sed` work in the same way, so I will cut it short here.
171
172As an example, a few days ago I wanted to add a tab to every line
173of a snippet of code, except for the first one. I used this:
174
175```
176$ sed '2,$ s/^/TAB/'
177```
178
179With a literal tab character (by pressing `Ctrl+V Ctrl+TAB`) instead
180of `TAB`. With GNU `sed` one can use `\t` instead.
181
182*(Recall that `^` means "the beginning of a line", so the command
183above inserts `TAB` at the beginning of each line from the second
184one to the last.)*
185
186### More commands
187
188With `sed`, one can do more than just find & replace. Here are
189some of its other (simple) commands:
190
191**Delete**: `d`. You can use it on a range of lines, the default being
192every line. Unexpectedly useful trick: you can use `| sed 'd'` instead of
193`> /dev/null' to suppress all standard output!
194
195**Change**: `c`. The syntax is a bit different from what we have seen
196so far. For example, to replace every line that ends with `0` or `5`
197with `bar` you can use
198
199```
200$ sed '/[05]$/ c\
201bar
202'
203```
204
205Notice the newline before and after `bar`.
206
207The `c` command also behaves a bit differently from other commands
208when given a range of addresses, because it replaces the whole range
209instead of operating on each addressed line one by one.
210
211**Insert**: `i`. The syntax is the same as for the `c` command, but
212text is just inserted, without deleting the current line.
213
214**Print**: `p`. Lines are printed by default, but if you use the `-n`
215option they are not. Useless trick: `sed -n '/RE/p'` is equivalent to
216`grep 'RE'`!
217
218**Quit**: `q`. This can be used to terminate sed earlier instead of,
219for example, piping its result or its input through `head`. But it is
220mostly known for the meme "`head` is
221[harmful](https://harmful.cat-v.org/software/), use `sed 11q` instead".
222
223## Advanced sed
224
225So far I have only described "simple" `sed` commands that operate line
226by line. These was pretty much all I knew about `sed` before writing
227this post. But then I found out that there are more advanced features,
228and I think they are worth mentioning.
229
230### Pattern space and hold space
231
232Reading the OpenBSD manual page, right after the general description
233of how `sed` works, you can read the following sentence:
234
235```
236Some of the functions use a hold space to save all or part of the pattern
237space for subsequent retrieval.
238```
239
240So, let's see how this *hold space* works.
241
242There are 5 commands that manipulate or otherwise use the hold space:
243`g`, `G`, `h`, `H` and `x`. The command `g` replaces the contents of the
244pattern space with that of the hold space, while `G` appends the hold
245space to the pattern space (with a newline character in between). The
246commands `h` and `H` do the same, but in the other direction (pattern
247space to hold space); you can memorize them as the initials of "hold"
248and "get". Finally, `x` swaps the contents of the two spaces.
249
250Ok, let's see an example. It's a bit hard for me to come up with a
251concrete one because I have never used this feature, so let's try
252a "puzzle example". Say you want to replace every empty line of a file
253with the content of the last line that started with a `>` character.
254
255For example, if you input this text:
256
257```
258> To avoid edge cases, say the first line alway starts with >
259This is
260a paragraph
261
262Another paragraph
263
264> Now use this line
265After this line
266
267> Ok now this
268> Actually, this
269
270The end.
271```
272
273You want to obtain:
274
275```
276> To avoid edge cases, say the first line alway starts with >
277This is
278a paragraph
279> To avoid edge cases, say the first line alway starts with >
280Another paragraph
281> To avoid edge cases, say the first line alway starts with >
282> Now use this line
283After this line
284> Now use this line
285> Ok now this
286> Actually, this
287> Actually, this
288The end.
289```
290
291To do this, you can use the following command:
292
293```
294$ sed `/^>/h; /^$/g'
295```
296
297As a reminder: We are using regular expressions to specify address;
298`^` matches the beginning of a line and `$` matches the end of a line,
299so `^$` matches a blank line.
300
301Yeah, this specific example is quite useless. Do you have any better
302example of use of the hold space in `sed`? Let me know!
303
304### Branching
305
306I'll cover this very briefly because, like for the previous part about
307the hold space, I have never used it in practice.
308
309If you are writing a longer `sed` script, you may be interested in
310(conditionally) jumping to different parts of your code. To do this,
311you can set a label with with `: label` and branch to it with `b label`.
312You can jump to a `label` conditionally, depending whether there has
313been a text substitution or not since last reading an input line, using
314`t label`.
315
316As an example: say you want to replace some text, but also add some
317kind of log of your work - for example, a line of text explaining that
318a replacement happened. Then you can do something like this:
319
320```
321$ sed 's/dog/cat/g; t log; b end; : log; { i\
322! At least one substitution was performed in the next line:
323}; : end'
324```
325
326In the code above we set two labels, `log` just before the command
327that adds the log line and `end` at the end of the `sed` script. If a
328substitution happens, we jump to `log`; if we do not jump to `log`,
329then next instruction makes us jump directly to the `end`. Kinda like
330programming with `goto`s!
331
332In this example I had to wrap the `i` command in curly braces `{}`,
333otherwise the semicolon needed to separate it from `: end` command would
334have been treated as part of the text to be inserted.
335
336## BSD sed vs GNU sed
337
338To conclude this post, I would like to highlight some of the differences
339between the
340[GNU implementation of `sed`](https://www.gnu.org/software/sed/manual/sed.html),
341which is found in most Linux distros except
342[Alpine](https://alpinelinux.org) and a few others, and the BSD version
343found in many
344[BSD operating systems](https://en.wikipedia.org/wiki/Berkeley_Software_Distribution),
345including MacOS. I am not sure all the BSD versions have the same features,
346but the main points discussed in this section should hold for all of them.
347
348Those listed below are all the differences I know of. If you know
349more, feel free to send me an email and I'll add them here!
350
351### BSD sed is more minimal
352
353In general BSD sed is more barebones, offering little more than POSIX mandates.
354If something can be done with BSD `sed` it can also be done with
355the GNU version, but the converse is not always true.
356
357GNU `sed` has some extra options, some more commands and an alternative
358syntax for some of the commands we have seen in this post - such as `c`
359and `i`. See
360[the Extended Commands section](https://www.gnu.org/software/sed/manual/sed.html#Extended-Commands)
361of the GNU manual for details.
362
363### Escape sequences
364
365In GNU `sed` one can use escape sequences such as `\n` and `\t` not
366only in regular expressions, but also in text - for example, in the
367replacement part of an `s` command. In BSD `sed`, this is not possible:
368one must insert literal special characters in their command - for example
369by pressing `Ctrl+V Ctrl+TAB` or by breaking a command with a newline,
370which is a bit ugly in my opinion.
371
372Escape sequences can be used in regular expressions in both the GNU
373and in the BSD version, see the section **Sed Regular Expressions** in the
374[OpenBSD](https://man.openbsd.org/sed)
375or
376[FreeBSD](https://man.freebsd.org/cgi/man.cgi?query=sed&apropos=0&sektion=0&manpath=FreeBSD+14.0-RELEASE+and+Ports&arch=default&format=html)
377manual pages for details.
378
379### Regular expression special syntax
380
381Both versions of `sed` let you choose between basic and extended regular
382expressions with the `-E` (or `-r`) flag, but the GNU version offers
383some new sets of characters not present in BSD.
384
385We have already seen `\b` (word boundary); others include `\w` (word characters,
386i.e. letters, digits or underscores) and `\s` (whitespace). See
387[the GNU manual](https://www.gnu.org/software/sed/manual/sed.html#regexp-extensions)
388for a full list.
389
390## Until next time... sort of
391
392It took me a long time to write this, but I am personally quite happy
393with the result. This is not a complete `sed` tutorial by any means,
394and the set of examples is not as comprehensive as the interested reader
395might like, but I think it is a decent overview.
396
397The next post in the series is supposed to be about `awk`, but I decided
398to take a small detour and talk about some other simple, special-purpose
399text filtering commands, such as `tr`, `head`, `fmt` and so on. Expect
400some short posts in this series before part 3 - after all, there are
401[uncountably many](https://en.wikipedia.org/wiki/Uncountable_set)
402numbers between two and 3!
diff --git a/src/series/series.md b/src/series/series.md
index 75bc301..34f34d7 100644
--- a/src/series/series.md
+++ b/src/series/series.md
@@ -1,7 +1,7 @@
1# List of blog series 1# List of blog series
2 2
3I my [blog](../blog) I sometimes write multiple posts on the same 3In my [blog](../blog) I sometimes write multiple posts on the same topic,
4topic, or posts in multiple parts. This page contains the list of these 4or I write posts in multiple parts. This page contains the list of these
5"blog series". 5"blog series".
6 6
7**Note:** This is *not* a full list of my blog posts divided by topic. 7**Note:** This is *not* a full list of my blog posts divided by topic.
@@ -26,9 +26,10 @@ Each post is introduced by a fictional context.
26In this series I explore three classic UNIX commands, in increasing order 26In this series I explore three classic UNIX commands, in increasing order
27of complexity: `grep`, `sed` and `awk`. Work in progress. 27of complexity: `grep`, `sed` and `awk`. Work in progress.
28 28
29* [grep](../blog/2023-08-20-grep) 29* Part 0: [Regular expressions](../blog/2023-06-16-regex)
30* sed [coming soon] 30* Part 1: [grep](../blog/2023-08-20-grep)
31* awk [coming less soon] 31* Part 2: [sed](../blog/2023-12-03-sed)
32* Part 3: awk (coming "soon")
32 33
33## The UNIX shell as an IDE 34## The UNIX shell as an IDE
34 35

Generated with cgit - Back to sebastiano.tronto.net