commit 1aadd39937cf9b0688ab680efe41faca471f459b
parent 34587f1c10d5fe0fe32b52f5032665ae8ce87d1a
Author: Sebastiano Tronto <sebastiano@tronto.net>
Date: Tue, 11 Jul 2023 23:21:25 +0200
Added blog post
Diffstat:
3 files changed, 348 insertions(+), 0 deletions(-)
diff --git a/src/blog/2023-07-11-feed/feed.md b/src/blog/2023-07-11-feed/feed.md
@@ -0,0 +1,340 @@
+# My minimalistic RSS feed setup
+
+A couple of years ago I started using
+[RSS](https://en.wikipedia.org/wiki/Rss)
+(or [atom](https://en.wikipedia.org/wiki/Atom_(standard)))
+feeds to stay up to date with websites and blogs I wanted to read.
+This method is more convenient than what I used before (i.e. open
+Firefox and open each website I want to follow in a new tab, one
+by one), but unfortunately not every website provides an RSS feed
+these days.
+
+At first I used [newsboat](https://newsboat.org), but I soon started
+disliking the curses interface - see also my rant on curses at the
+end of [this other blog post](../2022-12-24-ed). Then I discovered
+`sfeed`.
+
+## sfeed
+
+[`sfeed`](https://codemadness.org/sfeed-simple-feed-parser.html)
+is an extremely minimalistic RSS and atom reader: it reads
+the xml content of feed file from standard input and it outputs one line per
+feed item, with tab-separated timestamps, title, link and so on. This tool
+comes bundled with other commands that can be combined with it, such as
+`sfeed_plain`, which converts the output of sfeed into something
+more readable:
+
+```
+$ curl -L https://sebastiano.tronto.net/blog/feed.xml | sfeed | sfeed_plain
+ 2023-06-16 02:00 UNIX text filters, part 0 of 3: regular expressions https://sebastiano.tronto.net/blog/2023-06-16-regex
+ 2023-05-05 02:00 I had to debug C code on a smartphone https://sebastiano.tronto.net/blog/2023-05-05-debug-smartphone
+ 2023-04-10 02:00 The big rewrite https://sebastiano.tronto.net/blog/2023-04-10-the-big-rewrite
+ 2023-03-30 02:00 The man page reading club: dc(1) https://sebastiano.tronto.net/blog/2023-03-30-dc
+ 2023-03-06 01:00 Resizing my website's pictures with ImageMagick and find(1) https://sebastiano.tronto.net/blog/2023-03-06-resize-pictures
+...
+```
+
+One can also write a configuration file with all the desired feeds
+and fetch them with `sfeed_update`, or even use the `sfeed_curses`
+UI. But the reasons I tried out `sfeed` in the first place is that
+I *did not* want to use a curses UI, so I decided to stick with
+`sfeed_plain`.
+
+## My wrapper script - old versions
+
+In the project's homepage the following short script is presented to
+demonstrate the flexibility of sfeed:
+
+```
+#!/bin/sh
+url=$(sfeed_plain "$HOME/.sfeed/feeds/"* | dmenu -l 35 -i | \
+ sed -n 's@^.* \([a-zA-Z]*://\)\(.*\)$@\1\2@p')
+test -n "${url}" && $BROWSER "${url}"
+```
+
+The first line shows a list of feed items in
+[dmenu](https://tools.suckless.org/dmenu)
+to let the user select one, the second line opens the selected item
+in a web browser. I was impressed by how simple and clever this
+example was, and I decided to expand on it to build "my own" feed
+reader UI.
+
+In the first version I made, my feeds were separated in folders,
+one per file, and one could select multiple feeds or even entire
+folders via dmenu using
+[dmenu-filepicker](https://git.tronto.net/scripts/file/dmenu-filepicker.html)
+for file selection.
+Once the session was terminated, all shown feeds were marked as
+"read" by writing the timestamp of the last read item on a cache
+file, and they were not shown again on successive calls.
+
+This system worked fine for me, but at some point I grew tired of
+feeds being marked as "read" automatically. I also disliked the
+complexity of my own script. So I rewrote it from scratch, giving
+up the idea of marking feeds as read. This second version can still
+be found in the *old* folder of my
+[scripts repo](https://git.tronto.net/scripts), but I may remove it
+in the future. You will still be able to find it in the git history.
+
+I have happily used this second version for more than a year, but
+I had some minor issues with it. The main one was that, as I started
+adding more and more websites to my feed list, fetching them took
+longer and longer - up to 20-30 seconds; while the feed was loading,
+I could not start doing other stuff, because later dmenu would have
+grapped my keyboard while I was typing. Moreover, having a way to
+filter out old feed items is kinda useful when you check your feed
+relatively often. A few weeks ago I had enough and I decided to
+rewrite my wrapper script once again.
+
+## My wrapper script - current version
+
+In its current version, my `feed` scripts accepts four sub-commands:
+`get` to update the feed, `menu` to prompt a dmenu selection, `clear`
+to remove the old items and `show` to list all the new items.
+Since `clear` is a separate action, I do not have the problem I
+used to have with my first version, i.e. that feeds are automatically
+marked as read even if I sometimes do not want them to be.
+
+Let's walk through my last iteration on this script - you can find
+it in my scripts repository, but I'll include it at the end of this
+section too.
+
+At first I define some variables (mostly filenames), so that I can
+easily adapt the script if one day I want to move stuff around:
+
+```
+dir=$HOME/box/sfeed
+feeddir=$dir/urls
+destdir=$dir/new
+olddir=$dir/old
+readdir=$dir/last
+menu="dmenu -l 20 -i"
+urlopener=open-url
+```
+
+Here `open-url` is another one of my utility scripts.
+
+To update the feed, I loop over the files in my feed folder. Each
+file contains a single line with the feed's url, and the name of
+the file is the name / title of the website. The results of `sfeed`
+are piped into `sfeed_plain` and then saved to a file, and the most
+recent time stamp for each feed is updated.
+
+```
+getnew() {
+ for f in "$feeddir"/*; do
+ read -r url < "$f"
+ name=$(basename "$f")
+ d="$destdir/$name"
+ r="$readdir/$name"
+
+ [ -f "$r" ] && read -r lr < "$r" || lr=0
+
+ # Get new feed items
+ tmp=$(mktemp)
+ curl -s "$url" | sfeed | \
+ awk -v lr="$lr" '$1 > lr {print $0}' | \
+ tee "$tmp" | sfeed_plain >> "$d"
+
+ # Update last time stamp
+ awk -v lr="$lr" '$1 > lr {lr=$1} END {print lr}' <"$tmp" >"$r"
+ done
+}
+```
+
+The next snippet is used to show the new feed items.
+The `for` loop could be replaced by a simple
+`cat "$destdir"/*`, but I also want to prepend each line with
+the name of the website.
+
+```
+show() {
+ for f in "$destdir"/*; do
+ ff=$(basename "$f")
+ if [ -s "$f" ]; then
+ while read -r line; do
+ printf '%20s %s\n' "$ff" "$line"
+ done < "$f"
+ fi
+ done
+}
+```
+
+Finally, the following one-liner can be used to prompt the user to
+select and open the desired items in a browser using dmenu:
+
+```
+selectmenu() {
+ $menu | awk '{print $NF}' | xargs $urlopener
+}
+```
+
+The "clear" action is a straightfortward file management routine,
+and the rest of the script is just shell boilerplate code to parse
+the command line options and sub-commands. Putting it all together,
+the script looks like this:
+
+```
+#!/bin/sh
+
+# RSS feed manager
+
+# Requires: sfeed, sfeed_plain (get), dmenu, open-url (menu)
+
+# Usage: feed [-m menu] [get|menu|clear|show]
+
+dir=$HOME/box/sfeed
+feeddir=$dir/urls
+destdir=$dir/new
+olddir=$dir/old
+readdir=$dir/last
+menu="dmenu -l 20 -i"
+urlopener=open-url
+
+usage() {
+ echo "Usage: feed [get|menu|clear|show]"
+}
+
+getnew() {
+ for f in "$feeddir"/*; do
+ read -r url < "$f"
+ name=$(basename "$f")
+ d="$destdir/$name"
+ r="$readdir/$name"
+
+ [ -f "$r" ] && read -r lr < "$r" || lr=0
+
+ # Get new feed items
+ tmp=$(mktemp)
+ curl -s "$url" | sfeed | \
+ awk -v lr="$lr" '$1 > lr {print $0}' | \
+ tee "$tmp" | sfeed_plain >> "$d"
+
+ # Update last time stamp
+ awk -v lr="$lr" '$1 > lr {lr=$1} END {print lr}' <"$tmp" >"$r"
+ done
+}
+
+show() {
+ for f in "$destdir"/*; do
+ ff=$(basename "$f")
+ if [ -s "$f" ]; then
+ while read -r line; do
+ printf '%20s %s\n' "$ff" "$line"
+ done < "$f"
+ fi
+ done
+}
+
+selectmenu() {
+ $menu | awk '{print $NF}' | xargs $urlopener
+}
+
+while getopts "m:" opt; do
+ case "$opt" in
+ m)
+ menu="$OPTARG"
+ ;;
+ *)
+ usage
+ exit 1
+ ;;
+ esac
+done
+
+shift $((OPTIND - 1))
+
+if [ -z "$1" ]; then
+ usage
+ exit 1
+fi
+
+case "$1" in
+ get)
+ getnew
+ countnew=$(cat "$destdir"/* | wc -l)
+ echo "$countnew new feed items"
+ ;;
+ menu)
+ show | selectmenu
+ ;;
+ clear)
+ d="$olddir/$(date +'%Y-%m-%d-%H-%M-%S')"
+ mkdir "$d"
+ mv "$destdir"/* "$d/"
+ ;;
+ show)
+ show
+ ;;
+ *)
+ usage
+ exit 1
+ ;;
+esac
+```
+
+I personally like this approach of taking a simple program that
+only uses standard output and standard input and wrapping it around
+a shell script to have it do exactly what I want. The bulk of the
+work is done the "black box" program, and the shell scripts glues
+it together with the "configuration" files (in this case, my feed
+folder) and presents the results to me, interactively (e.g. via
+dmenu) or otherwise.
+
+At this point my feed-comsumption workflow would be something like
+this: first I `feed get`, then I do other stuff while the feed loads
+and later, after a couple of minutes or so, I run a `feed show` or
+`feed menu`. This is still not ideal, because whenever I want to
+check my feeds I still have to wait for them to be downloaded. The
+only way to go around it would be to have `feed get` run automatically
+when I am not thinking about it...
+
+## Setting up a cron job
+
+My personal laptop is not always connected to the internet, and in
+general I do not like having too many network-related jobs running
+in the background. But I do have a machine that is always connected
+to the internet: the VM instance hosting this website.
+
+Since my new setup saves my feed updates to local files, I can have
+a [cron job](https://en.wikipedia.org/wiki/Cron_job) fetch the new
+items and update files in a folder sync'd via
+[syncthing](https://syncthing.net) (yes, I do have that *one* network
+service constantly running in the background...). This setup is
+similar to the one I use to [fetch my email](../2022-10-19-email-setup).
+
+I rarely use cron, and I am always a little intimitaded by its
+syntax. But in the end to have `feed get` run every hour I just
+needed to add the following two lines via `crontab -e`:
+
+```
+MAILTO=""
+0 * * * * feed get
+```
+
+This is my definitive new setup, and I like it. It also has the
+advantage that I only need to install `sfeed` on my server and not
+locally, though I prefer to still keep it around.
+
+So far I have found one little caveat: if my feed gets updated after
+I read it and before I run a `feed clear`, some items may be deleted
+before I see them. This is easilly worked around by running a quick
+`feed show` before I clear the feeds up, but it is still worth
+keeping in mind.
+
+## Conclusions
+
+This is a summary of my last script-crafting adventure. As I was
+writing this post I realized I could probably use `sfeed_update`
+to simplify the script a bit, since I do not separate feeds into
+folders anymore. I have also found out that `sfeed_mbox` was created
+(at least I *think* it was not there the last time I checked) and I
+could use it to browse my feed with a mail client - see also
+[this video tutorial](https://josephchoe.com/rss-terminal) for a demo.
+
+With all of this, did I solve my problem in the best possible way?
+Definitely not. But does it work for me? Absolutely! Did I learn
+something new while doing this? Kind of, but mostly I have just
+excercised skills that I already had.
+
+All in all, it was a fun exercise.
diff --git a/src/blog/blog.md b/src/blog/blog.md
@@ -5,6 +5,7 @@
## 2023
+* 2023-07-11 [My minimalistic RSS feed setup](2023-07-11-feed)
* 2023-06-16 [UNIX text filters, part 0 of 3: regular expressions](2023-06-16-regex)
* 2023-05-05 [I had to debug C code on a smartphone](2023-05-05-debug-smartphone)
* 2023-04-10 [The big rewrite](2023-04-10-the-big-rewrite)
diff --git a/src/blog/feed.xml b/src/blog/feed.xml
@@ -9,6 +9,13 @@ Thoughts about software, computers and whatever I feel like sharing
</description>
<item>
+<title>My minimalistic RSS feed setup</title>
+<link>https://sebastiano.tronto.net/blog/2023-07-11-feed</link>
+<description>My minimalistic RSS feed setup</description>
+<pubDate>2023-07-11</pubDate>
+</item>
+
+<item>
<title>UNIX text filters, part 0 of 3: regular expressions</title>
<link>https://sebastiano.tronto.net/blog/2023-06-16-regex</link>
<description>UNIX text filters, part 0 of 3: regular expressions</description>
