path: root/src/blog/2023-07-11-feed/feed.md

# My minimalistic RSS feed setup

A couple of years ago I started using
[RSS](https://en.wikipedia.org/wiki/Rss)
(or [atom](https://en.wikipedia.org/wiki/Atom_(standard)))
feeds to stay up to date with websites and blogs I wanted to read.
This method is more convenient than what I used before (i.e. open
Firefox and open each website I want to follow in a new tab, one
by one), but unfortunately not every website provides an RSS feed
these days.

At first I used [newsboat](https://newsboat.org), but I soon started
disliking the curses interface - see also my rant on curses at the
end of [this other blog post](../2022-12-24-ed). Then I discovered
`sfeed`.

## sfeed

[`sfeed`](https://codemadness.org/sfeed-simple-feed-parser.html)
is an extremely minimalistic RSS and atom reader: it reads
the xml content of feed file from standard input and it outputs one line per
feed item, with tab-separated timestamps, title, link and so on. This tool
comes bundled with other commands that can be combined with it, such as
`sfeed_plain`, which converts the output of sfeed into something
more readable:

```
$ curl -L https://sebastiano.tronto.net/blog/feed.xml | sfeed | sfeed_plain
  2023-06-16 02:00  UNIX text filters, part 0 of 3: regular expressions                    https://sebastiano.tronto.net/blog/2023-06-16-regex
  2023-05-05 02:00  I had to debug C code on a smartphone                                  https://sebastiano.tronto.net/blog/2023-05-05-debug-smartphone
  2023-04-10 02:00  The big rewrite                                                        https://sebastiano.tronto.net/blog/2023-04-10-the-big-rewrite
  2023-03-30 02:00  The man page reading club: dc(1)                                       https://sebastiano.tronto.net/blog/2023-03-30-dc
  2023-03-06 01:00  Resizing my website's pictures with ImageMagick and find(1)            https://sebastiano.tronto.net/blog/2023-03-06-resize-pictures
...
```

One can also write a configuration file with all the desired feeds
and fetch them with `sfeed_update`, or even use the `sfeed_curses`
UI. But the reasons I tried out `sfeed` in the first place is that
I *did not* want to use a curses UI, so I decided to stick with
`sfeed_plain`.

## My wrapper script - old versions

In the project's homepage the following short script is presented to
demonstrate the flexibility of sfeed:

```
#!/bin/sh
url=$(sfeed_plain "$HOME/.sfeed/feeds/"* | dmenu -l 35 -i | \
	sed -n 's@^.* \([a-zA-Z]*://\)\(.*\)$@\1\2@p')
test -n "${url}" && $BROWSER "${url}"
```

The first line shows a list of feed items in
[dmenu](https://tools.suckless.org/dmenu)
to let the user select one, the second line opens the selected item
in a web browser. I was impressed by how simple and clever this
example was, and I decided to expand on it to build "my own" feed
reader UI.

In the first version I made, my feeds were separated in folders,
one per file, and one could select multiple feeds or even entire
folders via dmenu using
[dmenu-filepicker](https://git.tronto.net/scripts/tree/dmenu-filepicker)
for file selection.
Once the session was terminated, all shown feeds were marked as
"read" by writing the timestamp of the last read item on a cache
file, and they were not shown again on successive calls.

This system worked fine for me, but at some point I grew tired of
feeds being marked as "read" automatically. I also disliked the
complexity of my own script.  So I rewrote it from scratch, giving
up the idea of marking feeds as read. This second version can still
be found in the *old* folder of my
[scripts repo](https://git.tronto.net/scripts), but I may remove it
in the future. You will still be able to find it in the git history.

I have happily used this second version for more than a year, but
I had some minor issues with it. The main one was that, as I started
adding more and more websites to my feed list, fetching them took
longer and longer - up to 20-30 seconds; while the feed was loading,
I could not start doing other stuff, because later dmenu would have
grapped my keyboard while I was typing. Moreover, having a way to
filter out old feed items is kinda useful when you check your feed
relatively often.  A few weeks ago I had enough and I decided to
rewrite my wrapper script once again.

## My wrapper script - current version

In its current version, my `feed` scripts accepts four sub-commands:
`get` to update the feed, `menu` to prompt a dmenu selection, `clear`
to remove the old items and `show` to list all the new items.
Since `clear` is a separate action, I do not have the problem I
used to have with my first version, i.e. that feeds are automatically
marked as read even if I sometimes do not want them to be.

Let's walk through my last iteration on this script - you can find
it in my scripts repository, but I'll include it at the end of this
section too.

At first I define some variables (mostly filenames), so that I can
easily adapt the script if one day I want to move stuff around:

```
dir=$HOME/box/sfeed
feeddir=$dir/urls
destdir=$dir/new
olddir=$dir/old
readdir=$dir/last
menu="dmenu -l 20 -i"
urlopener=open-url
```

Here `open-url` is another one of my utility scripts.

To update the feed, I loop over the files in my feed folder.  Each
file contains a single line with the feed's url, and the name of
the file is the name / title of the website. The results of `sfeed`
are piped into `sfeed_plain` and then saved to a file, and the most
recent time stamp for each feed is updated.

```
getnew() {
	for f in "$feeddir"/*; do
		read -r url < "$f"
		name=$(basename "$f")
		d="$destdir/$name"
		r="$readdir/$name"

		[ -f "$r" ] && read -r lr < "$r" || lr=0

		# Get new feed items
		tmp=$(mktemp)
		curl -s "$url" | sfeed | \
		awk -v lr="$lr" '$1 > lr {print $0}' | \
		tee "$tmp" | sfeed_plain >> "$d"

		# Update last time stamp
		awk -v lr="$lr" '$1 > lr {lr=$1} END {print lr}' <"$tmp" >"$r"
	done
}
```

The next snippet is used to show the new feed items.
The `for` loop could be replaced by a simple
`cat "$destdir"/*`, but I also want to prepend each line with 
the name of the website.

```
show() {
	for f in "$destdir"/*; do
		ff=$(basename "$f")
		if [ -s "$f" ]; then
			while read -r line; do
				printf '%20s    %s\n' "$ff" "$line"
			done < "$f"
		fi
	done
}
```

Finally, the following one-liner can be used to prompt the user to
select and open the desired items in a browser using dmenu:

```
selectmenu() {
	$menu | awk '{print $NF}' | xargs $urlopener
}
```

The "clear" action is a straightfortward file management routine,
and the rest of the script is just shell boilerplate code to parse
the command line options and sub-commands. Putting it all together,
the script looks like this:

```
#!/bin/sh

# RSS feed manager

# Requires: sfeed, sfeed_plain (get), dmenu, open-url (menu)

# Usage: feed [-m menu] [get|menu|clear|show]

dir=$HOME/box/sfeed
feeddir=$dir/urls
destdir=$dir/new
olddir=$dir/old
readdir=$dir/last
menu="dmenu -l 20 -i"
urlopener=open-url

usage() {
	echo "Usage: feed [get|menu|clear|show]"
}

getnew() {
	for f in "$feeddir"/*; do
		read -r url < "$f"
		name=$(basename "$f")
		d="$destdir/$name"
		r="$readdir/$name"

		[ -f "$r" ] && read -r lr < "$r" || lr=0

		# Get new feed items
		tmp=$(mktemp)
		curl -s "$url" | sfeed | \
		awk -v lr="$lr" '$1 > lr {print $0}' | \
		tee "$tmp" | sfeed_plain >> "$d"

		# Update last time stamp
		awk -v lr="$lr" '$1 > lr {lr=$1} END {print lr}' <"$tmp" >"$r"
	done
}

show() {
	for f in "$destdir"/*; do
		ff=$(basename "$f")
		if [ -s "$f" ]; then
			while read -r line; do
				printf '%20s    %s\n' "$ff" "$line"
			done < "$f"
		fi
	done
}

selectmenu() {
	$menu | awk '{print $NF}' | xargs $urlopener
}

while getopts "m:" opt; do
	case "$opt" in
		m)
			menu="$OPTARG"
			;;
		*)
			usage
			exit 1
			;;
	esac
done

shift $((OPTIND - 1))

if [ -z "$1" ]; then
	usage
	exit 1
fi

case "$1" in
	get)
		getnew
		countnew=$(cat "$destdir"/* | wc -l)
		echo "$countnew new feed items"
		;;
	menu)
		show | selectmenu
		;;
	clear)
		d="$olddir/$(date +'%Y-%m-%d-%H-%M-%S')"
		mkdir "$d"
		mv "$destdir"/* "$d/"
		;;
	show)
		show
		;;
	*)
		usage
		exit 1
		;;
esac
```

I personally like this approach of taking a simple program that
only uses standard output and standard input and wrapping it around
a shell script to have it do exactly what I want. The bulk of the
work is done the "black box" program, and the shell scripts glues
it together with the "configuration" files (in this case, my feed
folder) and presents the results to me, interactively (e.g. via
dmenu) or otherwise.

At this point my feed-comsumption workflow would be something like
this: first I `feed get`, then I do other stuff while the feed loads
and later, after a couple of minutes or so, I run a `feed show` or
`feed menu`.  This is still not ideal, because whenever I want to
check my feeds I still have to wait for them to be downloaded.  The
only way to go around it would be to have `feed get` run automatically
when I am not thinking about it...

## Setting up a cron job

My personal laptop is not always connected to the internet, and in
general I do not like having too many network-related jobs running
in the background.  But I do have a machine that is always connected
to the internet: the VM instance hosting this website.

Since my new setup saves my feed updates to local files, I can have
a [cron job](https://en.wikipedia.org/wiki/Cron_job) fetch the new
items and update files in a folder sync'd via
[syncthing](https://syncthing.net) (yes, I do have that *one* network
service constantly running in the background...). This setup is
similar to the one I use to [fetch my email](../2022-10-19-email-setup).

I rarely use cron, and I am always a little intimitaded by its
syntax. But in the end to have `feed get` run every hour I just
needed to add the following two lines via `crontab -e`:

```
MAILTO=""
0 * * * * feed get
```

This is my definitive new setup, and I like it. It also has the
advantage that I only need to install `sfeed` on my server and not
locally, though I prefer to still keep it around.

So far I have found one little caveat: if my feed gets updated after
I read it and before I run a `feed clear`, some items may be deleted
before I see them.  This is easilly worked around by running a quick
`feed show` before I clear the feeds up, but it is still worth
keeping in mind.

## Conclusions

This is a summary of my last script-crafting adventure. As I was
writing this post I realized I could probably use `sfeed_update`
to simplify the script a bit, since I do not separate feeds into
folders anymore. I have also found out that `sfeed_mbox` was created
(at least I *think* it was not there the last time I checked) and I
could use it to browse my feed with a mail client - see also
[this video tutorial](https://josephchoe.com/rss-terminal) for a demo.

With all of this, did I solve my problem in the best possible way?
Definitely not. But does it work for me? Absolutely! Did I learn
something new while doing this? Kind of, but mostly I have just
excercised skills that I already had.

All in all, it was a fun exercise.