# UNIX text filters, part 1 of 3: grep After [the preliminary post on regular expressions](../2023-06-16-regex), we are ready to begin this series on *text filters*. This time we'll explore `grep`, the most simple kind of filter: given a bunch of lines of text, print out only those that match a certain criterion. I will only describe a few basic options. All that I mention here is POSIX-standard, with the exception of the option `-o`. This means that the content of this post is valid in pretty much any UNIX-like OS, but check your manual pages before copy-pasting my code - I can always make mistakes. Without further ado, let's dive in! ## Standard usage If you are familiar with how (UNIX) programs read from standard output and write to standard output, the idea behing `grep` is easily explained: the command ``` $ grep PATTERN ``` will read lines from standard input and write to standard output only those that contain the given `PATTERN`. If you specify file names after the pattern ``` $ grep PATTERN file1 file2 ... ``` `grep` will read those files instead of standard input. The `PATTERN` can also be a [regular expression](../2023-06-16-regex). In other words, you can use `grep` to look for certain pieces of text in a file or in the output of another command. If you do not understand all of this is about, start reading from the **Examples** section below to get an idea. Now let's see how you can tune `grep`'s behavior to your needs. ### What to match: `-i`, `-v` A common use of `grep`, especially for non-programming tasks, is to look for occurrences of a specific word in a long text. In this case one usually does not care if the word is all lowercase or capitalized, for example because at the beginning of a sentence. If you find yourself in this situation, you can use the `-i` option to make `grep` case-insensitive. Sometimes it easier to spell out what you *do not* want to match - for example, say you want all non-empty lines of a given file. In this case you can use the `-v` option to invert the behavior of `grep`, such as: ``` $ grep -v "^$" file ``` Here `"^$"` is a regular expression that matches all lines where the beginning of the line (in regex language, `^`) is immediately followed by the end of the line (`$`); in other words, empty lines. ### More on patterns: `-E`, `-e`, `-F`, `-f` Up to now I have not specified what *kind* of regular expression `grep` uses. By default it uses basic regular expressions, but it uses extended regular expressions if called with the `-E` option. Equivalentrly, you can use the command `egrep`. If you want to turn off regular expressions altogether, you can use `grep -F` (or `fgrep`). If you want to select lines that match *any* of a number of patterns, you can use the `-e` option: ``` $ grep -e PATTERN1 -e PATTERN2 -e ... [file1 file2 ...] ``` Alternatively you can write your pattern in a file, one per line, and use: ``` $ grep -f PATTERN_FILE [file1 file2 ...] ``` ### Grepping multiple files: `-l`, `-n` Sometimes I use `grep` to find occurrences of a certain string in a bunch of files, for example with ``` $ grep "word" * ``` When used with multiple input files like this, `grep` will precede each output line with the name of the file that contains it. If the option `-n` is used, the line number is also shown. If `-l` is used, only the name of the file is shown, and each file is shown at most once. If you do not want to print the file names at all, you can always `cat` into `grep`: ``` $ cat file1 file2 ... | grep ``` But if anyone asks, you did not learn this from me - UUOC (Useless Use Of Cat) is a considered a crime in some circles. *Update 2023-09-02: I have just discovered that the the `-h` option can be used to hide the file names, so no need for piping cats. However, though present both in OpenBSD's and GNU's versions of `grep`, this option is not POSIX standard.* ### Matching only part of a line: `-o` You may not always want the *full line* containing a piece of text. Sometimes you just want a specific part of a line, and you know exactly how to match it with a regular expression. In this case you can use the `-o` option - we'll see an example below. The `-o` is not POSIX-standard. It is ubiquitous though, and it should be present in pretty much any version of `grep`. ## Examples Now that we now the basics, let's see some exciting applications of `grep`! Nah, I am kidding, they are not exciting. But they are useful. Boring, but useful. ### Filter command output Probably my first use of `grep` was to filter out irrelevant part of some command's output. Say for example you are troubleshooting a problem with your webcam: you can use `dmesg` to check what your operating system knows about it, but most of the output is useless to your specific problem. No worries, you can pipe `dmesg` into `grep`: ``` $ dmesg | grep video acpivideo0 at acpi0: VGA_ acpivout0 at acpivideo0: LCDD uvideo0 at uhub0 port 6 configuration 1 interface 0 "JMICRON TECHNOLOGIES CO., LTD. USB2.0 UVC VGA WebCam" rev 2.00/2.04 addr 2 video0 at uvideo0 ``` ### Look stuff up in files Sometimes you may want to search something in a bunch of files. Let's say for example I want to check in which of my old blog posts I have mentioned "Linux": ``` $ grep -l Linux src/blog/*/* src/blog/2022-05-29-man/man.md src/blog/2022-08-14-website/website.md src/blog/2022-09-10-netbooks/netbooks.md src/blog/2023-01-28-windows-desktop/windows-desktop.md src/blog/2023-02-25-job-control/job-control.md src/blog/2023-02-25-job-control/jobs-diagram.pdf ``` Or say I am working on one of my software projects, and I do not remember where a certain function is defined: ``` $ grep -n "^apply_move(" src/*.c src/moves.c:206:apply_move(Move m, Cube cube) ``` *Note: the command above works because, when I write C code, I write function names on a newline. See also [this older post](../2022-06-12-shell-ide-sed) for another example that takes advantage of this, this time using `sed`.* ### Grepping URLs Looking for URLs in a piece of text is a common enough operation for me that I saved it into a [script](https://git.tronto.net/scripts) for ease of use, that I called `urlgrep`. URLs can be complicated, so for a long time I used a regular expression copied from somewhere on the internet. Now now that I am more familiar with `grep` and regular expressions, I have written my own - it does not work perfectly, but at least I understand it and I can keep tweaking it if I find errors. Let's build it together! What does a URL look like? It usually starts with either a *protocol* followed by a colon, or with `www.`. Then a bunch of valid characters follow. There are probably more rules to it, but to keep is simple we can start like this (using *extended* regular expressions): ``` regex="(($protocols):|www\.)[$valid_chars]+" ``` For protocols we can use ``` protocols='http|https|ftp|sftp|gemini|mailto' ``` I have thrown `mailto` in there because it is quite common in links web pages. The valid characters are: ``` valid_chars="][a-zA-Z0-9_~/?#@!$&'()*+=.,;:-" ``` (Yes, these ones I actually copied somewhere online). Finally we can find all URLs with ``` $ egrep -o "$regex" ``` As I mentioned above there are some problems with this. For example if a URL is not terminated by a space, the characters following it may be grepped too. For example: ``` $ urlgrep