commit 1b69fbdce80f4ee2d46be63be34b4cfb4e4e11d7
parent b473f7ce3895eccaf7691f6fc9296cb3c695a189
Author: Sebastiano Tronto <sebastiano@tronto.net>
Date: Thu, 24 Nov 2022 00:01:47 +0100
Added blog post(s)
Diffstat:
4 files changed, 516 insertions(+), 0 deletions(-)
diff --git a/src/blog/2022-10-19-email-setup/email-setup.md b/src/blog/2022-10-19-email-setup/email-setup.md
@@ -0,0 +1,201 @@
+# Keeping my email sorted (the hard way)
+
+I have recently made some changes to my email setup. In this post I'll
+explain the motivation behind these changes and what I did in practice.
+
+## Self-hosting?
+
+When I got my virtual machine up and running at
+[openbsd.amsterdam](https://openbsd.amsterdam/) - the one where this
+website is hosted - I originally planned to host my private email
+server there too. I knew this was probably a hard task, but you know,
+everything is hard until you learn how to do it.
+
+I wanted to do this for a couple of reasons. The main one was to use my
+`@tronto.net` email address, but I also liked the idea of staying away
+from large internet companies (my main email address was connected to my
+Google account). Not that there is anything inherently wrong with using
+services from this big companies, but I like the idea of not being too
+dependent on them.
+
+After reading
+[a nice tutorial at poolp.org](https://poolp.org/posts/2019-09-14/setting-up-a-mail-server-with-opensmtpd-dovecot-and-rspamd/)
+I was a bit discouraged. The guide was well-written, all the steps seemed
+doable if taken one by one, and I was happy to have dug into this topic
+because I learned a lot. However, an email server apparently consists of
+a lot of moving pieces: an smtp server, spam filter, DNS, DKIM... it is
+a lot to keep track of. Even assuming that I would be able to set this
+thing up AND to keep in mind what each of these pieces does, as soon as a
+problem of any kind arises - config-breaking updates? domain registration
+expiring? me messing up with my VM and making it unreachable? - I knew I
+had to be one to fix the mistake. And I cannot afford to be immediately
+available whenever something bad happens. Sometimes I might just have a
+full week were I don't have time to fiddle around with smtpd and whatnot,
+and I can't afford being unreachable via email for a week.
+
+## My old setup (until September 2022)
+
+Having abandoned the idea of self-hosting, I looked for alternatives. I
+figured that if my goals were just to use my own domain and stay away
+from Google, I could sign up for a smaller email provider that offers
+custom domains. It turns out there are a lot of them. After some careful
+considerations I decided to go with [mailbox.org](https://mailbox.org). I
+like their transparency and privacy focus and the fact that they are
+based in the EU. I pay 3€ per month (the 1€ tier does not offer
+custom domains) and I am happy with their service.
+
+Setting up the server side was quite simle. Using custom domains
+requires a tiny bit of work, but it was all well explained in the
+[FAQs](https://kb.mailbox.org/en/private/custom-domains).
+
+On my local machine I used (and still use) the amazing
+[mblaze](https://github.com/leahneukirchen/mblaze), which is essentially
+[MH](https://en.wikipedia.org/wiki/MH_Message_Handling_System)
+for [Maildir](https://en.wikipedia.org/wiki/Maildir) folders.
+In practice, mblaze is a set of commands to manage emails directly
+from the command line, without using a graphical environment or a
+[TUI](https://en.wikipedia.org/wiki/Text-based_user_interface) like
+[Mutt](https://en.wikipedia.org/wiki/Mutt_(email_client)). This system
+is incredibly flexible, check it out if you don't know it!
+
+Being just a mail user agent, mblaze cannot retrieve or send
+email. These tasks can be accomplished by other small pieces of
+software: I used [msmtp](https://marlam.de/msmtp) for sending
+email and [mpop](https://marlam.de/mpop) for downloading it
+from mailbox.org's server. As the name suggests, mpop uses the
+[POP3](https://en.wikipedia.org/wiki/Post_Office_Protocol)
+protocol instead of the more common
+[IMAP](https://en.wikipedia.org/wiki/Internet_Message_Access_Protocol).
+The main difference is that POP3 simply retrieves your email, while
+IMAP keeps the server and client folders synchronized. There are many
+advantages and disadvantages to this choice, I won't go into detail on
+them in this post.
+
+As for my other devices, my local mailfolder is kept in sync with my
+server using [syncthing](https://syncthing.net). I also use an amail
+client on my phone with IMAP, connected directly to the mail server.
+
+## Nitpicking
+
+Since I am subscribed to a couple of high-traffic mailing lists that
+I read just for curiosity, it is necessary for me to have an easy way
+to download and view regular emails separately from that coming from
+mailing lists.
+
+This was kinda easy to set up with mpop's filters, but my configuration
+was a bit of a hack. One disadvantage of this soution was that it
+only solved the problem on my laptop(s). On my webmail and on my phone,
+my inbox was a complete mess of mailing lists, newsletter and a few
+important emails.
+
+After thinking about it for a while I figured that an elegant solution
+would be to set up alternative email addresses for receiving mailing
+list emails, like `list@tronto.net`. Then I would manage those different
+mailboxes separately.
+
+Setting the aliases up on mailbox.org was easy, but unfortunately all my
+`@tronto.net` address used the same inbox, so I did not solve any problem
+at all. I could add some sub-folders and set up filters so that incoming
+mail gets sorted out, but the app on my phone could not read sub-folders
+and mailbox did not allow top-level folders (or I could not find a way to
+create them). Besides, tinkering with IMAP folders was not something
+that I found particularly exciting.
+
+But there was another solution...
+
+## My current setup (since September 2022)
+
+I decided to try and redirect the mailing list emails to my personal
+server. Configuring OpenBSD's smtp to receive emails from one specific
+outside source (my mailbox.org account) and sort them into some local
+folders is order of magnitudes easier than setting up a full-fledged
+email server. No problems with DKIM, no incoming spam, no nothing.
+
+It took me a few hours to figure our how to do this, but in the
+end it is just a matter of configuring a few filters on mailbox.org
+and adding a couple of lines to `/etc/mail/stmp.conf`.
+
+### mailbox.org filters
+
+On my mailbox.org webmail I simply set up a filter to redirect any email
+sent to `list@example.com` (a made-up name for the mailing list I am
+subscribed to) to my private server. No copy of these emails is kept on
+the server, so they don't clutter my normal (IMAP) inbox. I risk missing
+a few of these emails if my server goes down, but it is a public mailing
+list and I can always check the archives online.
+
+I could not just send these emails to `something@tronto.net`, otherwise
+they would simply be taken care of by mailbox - the MX records for my
+domain point to their servers. But it turns out you can send mail to
+a server using its IP address, as long as the server is configured to
+accept such mail. So I set up the redirects to `list@[46.23.91.214]` -
+where `46.23.91.214` is the IP address of my server.
+
+### smtpd.conf
+
+The second step is configuring smtpd, OpenBSD's default mail server
+daemon, to deal with incoming email.
+
+First of all we need to list the virtual user `list` in
+`/etc/mail/aliases` so that any mail sent to it is interpreted as being
+sent to my regular user.
+
+```
+# cat 'list: sebastiano' >> /etc/mail/aliases
+```
+
+Then we have to change the line `listen on lo0` to `listen on all`
+in `/etc/smtp.conf`.
+
+Then we need to add an `action` and a `match` lines to the same file:
+
+```
+# cat << EOF >> /etc/mail/smtpd.conf
+> action "list" maildir "~/mail/list" alias <aliases>
+> match from any for rcpt-to "list@[46.23.91.214]" action "list"
+> EOF
+```
+
+And finally restart smtpd with `rcctl restart smtpd`.
+
+This does the trick: now all email I receive from the `list@example.com`
+mailing list is redirected by my mailbox account to my private server,
+where smtpd takes care of sending it to the mail directory `~/mail/list`.
+
+### No mpop needed
+
+Once the mail is delivered to `~/mail/list`, I can get it from there
+to my laptop in any way I like - for example using syncthing, like I
+do for all my important files. In this way the mailing list emails are
+regularly downloaded and kept in sync, and I don't need to use mpop to
+retrieve them.
+
+This is quite convenient, one less piece of software to keep track of!
+In fact, I can do the same for all other email I receive. I just need
+to set up the appropriate rules on mailbox: this time I want the mail
+to be sent to `sebastiano@[46.23.91.214]` and *a copy to be kept on the
+mailbox.org server*, so that I can easily access it from my phone's app as
+well. Then I add two slightly different lines to `/etc/mail/smtpd.conf`:
+
+```
+# cat << EOF >> /etc/mail/smtpd.conf
+> action "seb" maildir "~/mail/inbox" user sebastiano
+> match from any for rcpt-to "sebastiano@[46.23.91.214]" action "seb"
+> EOF
+```
+
+And the new setup is ready!
+
+### Sending email
+
+I did not change the way I send email: I still use msmtp.
+
+## Happy now?
+
+Yes, this new setup works and I am always happy when things work.
+Of course, one might make the case that things worked before as well...
+
+I am happy that I could work my way around a basic smtpd configuration.
+Besides being useful knowledge on its own, it may make a second attempt
+at self-hosting my email less daunting. I don't know if I am ever going
+to try that, though.
diff --git a/src/blog/2022-11-23-git-host/git-host.md b/src/blog/2022-11-23-git-host/git-host.md
@@ -0,0 +1,299 @@
+# Self-hosted git pages with stagit (featuring ed, the standard editor)
+
+This is a follow-up to my earlier blog entry
+[How I update my website](../2022-08-14-website).
+
+If you work on one or more personal software projects as a hobby,
+chances are you are using `git`. To publish your project online
+you may be using a website like GitHub, or perhaps a more
+open source friendly alternative such as
+[GitLab](https://about.gitlab.com/) or [sourcehut](https://sourcehut.org/).
+
+But have you considered hosting your repositories, and serving them
+via web pages, on your personal server? In this post I am going to show
+you how I do it, in the usual minimalist and
+not-using-what-I-do-not-understand style.
+
+You can see the final result on my [git pages](https://git.tronto.net).
+The scripts and other files I use to set this up are accessible
+[here](https://git.tronto.net/git-hooks).
+
+## Hosting git repositories on your own server
+
+This step is quite simple, and you can just follow
+[Roman Zolotarev's tutorial](https://rgz.ee/git.html) like I did - adapting
+the first few steps to your OS if you are not running OpenBSD.
+
+To sum it up, you need to:
+
+1. Create a dedicate `git` account on your server (optional, if you know
+ what you are doing).
+
+2. Add your public SSH key to the `~/.ssh/authorized_keys` file of your newly
+ created remote account.
+
+3. Initialize a git repository on your server with `git init REPOSITORY`.
+
+4. Clone it via SSH with `git clone git@SERVER:REPOSITORY`.
+
+And you are done! If you want to use multiple remotes, for example your private
+server and GitHub, you can do so by adding a push-only URL with
+`git remote set-url --add --push origin URL`. But don't trust me on this
+exact command, I always have to look it up
+- you should do the same before running it.
+
+Now your repositories are online, but how can you make them browsable via web?
+
+## stagit
+
+The tool I use to serve my git repositories as static web pages is
+[stagit](https://codemadness.org/stagit.html). It is very easy to describe what
+stagit does: running it on a git reporitory produces some directories with
+a bunch of html files that you can simply move to your www hosting directory.
+
+After generating the pages you can personalize them by copying your logo,
+[favicon](https://en.wikipedia.org/wiki/Favicon) or CSS style sheet. You can
+use `stagit-index` to generate
+[an index page for your repositories](https://git.tronto.net/). Since everything
+consists of html files, you can simply edit them to personalize your git pages
+even further - and below you'll see some examples.
+
+But you definitely do not want to do this by hand every time you push a commit.
+Since the pages stagit generates are *static*, they do not update
+automatically: you'll have to run stagit again every time. You can automate
+this for example by running stagit periodically with cron, but there is an
+easier way:
+[git hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks).
+
+## Git hooks
+
+By saving suitably named executable files inside your project's
+`.git/hooks` directory you can automate any process you want during
+certain stages of your git workflow. For example you can use a `pre-commit`
+script to run commands before you commit changes, or a `pre-push` script
+to do something before pushing a commit.
+
+The hooks are divided in client-side and server-side. We are interested in
+the server-side `post-receive` hook, which is executed on the remote every
+time a commit is received. This is the last ingredient we need for our setup:
+a simple `post-receive` hook that runs stagit and copies the files it
+generates to the appropriate folder will do the trick.
+
+## My setup
+
+Throughout the rest of the post I will use the following variables, to be
+set at the beginning of the script:
+
+```
+yourname="Sebastiano Tronto" # Author's name
+repo="$PWD" # The full path of the repository
+name="$(basename $repo .git)" # The name of the repository
+baseurl="https://git.tronto.net" # The base URL for the repository
+basedir="/var/www/htdocs/git.tronto.net" # The base directory for www files
+htdir="$basedir/$name" # The www directory for the repository
+```
+
+### Basic stagit usage
+
+The first thing we want to do is to set up some basic information in the
+`owner` and `url` files that stagit is going to use. The hook is run in the
+repository's directory, so it is not necessary to specify a full path:
+
+```
+echo "$yourname" > owner
+echo "$baseurl/$name" > url
+```
+
+Next we prepare the target directory by removing old files and creating
+it if necessary:
+
+```
+rm -rf "$htdir"
+mkdir -p "$basedir"
+```
+
+To make the repository clonable by anyone from the same URL used to
+view it, we need to copy the whole directory to the
+www directory we have just created:
+
+```
+cp -r "$repo" "$htdir"
+```
+
+And finally we can run stagit:
+
+```
+cd "$htdir"
+stagit -l 100 "$repo"
+```
+
+The `-l` option is used to specify how many commits should be visibile
+in the log page.
+
+For some basic personalization we can choose a different default page (index
+file). I like to have the file list:
+
+```
+cp "$htdir/files.html" "$htdir/index.html"
+```
+
+And we can use our css style sheet, logo and icon:
+
+```
+cp $filesdir/favicon.png ./
+cp $filesdir/logo.png ./
+cp $filesdir/style.css ./
+```
+
+### Bells and whistles
+
+I like stagit's simplicity, but there are a couple of things that I want to
+add or change:
+
+* I would like every page to show a simple footer at the bottom of
+ each page.
+* I would like to have a download button so that people who don't use git
+ can still download my files. This makes sense especially for those
+ repos that are mostly documents, such as my
+ [lecture notes](https://git.tronto.net/mathsoftware) or my
+ [FMC tutorial](https://git.tronto.net/fmctutorial).
+* I would like to convert README.md files to html.
+
+If I were calling stagit by hand after each `git push`, I
+could simply make these changes with a text editor. But I want to automate
+this! How can we edit files in a shell script?
+
+Enter `ed`, [the standard editor](https://www.gnu.org/fun/jokes/ed-msg.html).
+`ed` is a [line text editor](https://en.wikipedia.org/wiki/Line_editor)
+initially released with UNIX Version 1. I am going to talk about it more
+extensively in the next episode of my
+[man page reading club](../2022-05-29-man/)
+series. Without going into detail, `ed` does not show you the text you
+are editing in a 2-dimensional windows: instead, it offers you a command
+line prompt that you can use to run editing commands, such as `a` to add
+text or `p` to print one or more lines of the file.
+
+This might seems like a totally cumbersome way of editing a file, but
+there is one nice side-effect: `ed` is completely scriptable. This means
+that if you know exactly what the file you want to edit looks like,
+you can write the commands you want to run in advance and feed them to
+the editor via standard input, instead of typing them interactively.
+This is exactly what we want to do!
+
+Going back to my stagit setup, say we have a file `bottom.html` that
+looks like this:
+
+```
+<hr class="line">
+<footer> <table>
+<tr> <td class="contact">
+ Back to <a href="https://sebastiano.tronto.net"> sebastiano.tronto.net </a>
+</td>
+<td class="hosted">
+ Generated with <a href="https://codemadness.org/stagit.html">stagit</a>
+</td> </tr>
+</table> </footer>
+```
+
+and we want to insert its content in the file `file.html`, before
+the line that contains the closing tag `</body>`. We can use the
+following one-liner:
+
+```
+printf '%s\n' "/<\/body>" i "$(cat bottom.html)" . w | ed -s file.html
+```
+
+Here the `printf` command is used to feed the tokens `/<\/body>`, `i`,
+`$(cat bottom.html)`, `.` and `w` to `ed`. These are going to
+be interpreted as: "search for the closing tag `</body>`;
+insert the following text until you encounter a single dot on a line:
+[contents of the file `bottom.html`] single dot; save."
+If this seems obscure, I suggest you read
+[`ed`'s manual page](https://man.openbsd.org/OpenBSD-7.2/ed), or wait
+for my next blog post!
+
+The command for adding the download button is similar, after we
+generate a zip archive of the repository using `git archive`:
+
+```
+git archive HEAD -o "$basedir/$name.zip"
+printf '%s\n' \
+ "/log\.html\">Log<\/a>" i \
+ "<a href=\"$baseurl/$name.zip\">Download</a> |" . w \
+| ed -s file.html
+```
+
+Here I am using backlashes to ignore the newline character, so that
+I can use more lines for readability.
+
+The two code snippets above have to be run for every html file generated
+by stagit. To loop over all these files, you can use `find`:
+
+```
+for f in $(find "$htdir" -name "*.html"); do
+ [stuff...]
+done
+```
+
+The command to turn README.md files into a formatted html page is a bit
+more complicated, but I will try to keep the explanation short, since
+this post is already quite long. Feel free to send me an email if you
+have questions!
+
+To have an idea of what the README.md.html file generated by
+stagit looks like, you can check out the html of
+[this page](https://codemadness.org/git/stagit/file/README.html),
+for example (right click and "View page source" or something similar in
+most browsers, or `curl [URL]` if you are cool).
+
+First, since I am using bare git repositories, I need to actually "create"
+the original README.md file - instead of using its rendered-as-plain-text
+html version generated by stagit - using `git show`. Then we need to
+remove from README.md.html all the lines that
+are part of the code listing, i.e. all those that contain a `class="line"`
+string. The `ed` command to do this is `g/class=\"line\"/d`. Then we
+need to remove a couple more lines and finally we can insert the result
+of the command `lowdown file/README.md`, which converts the markdown
+file to html, into the correct place. The final result is:
+
+```
+git show master:README.md > file/README.md
+printf '%s\n' \
+ g/class=\"line\"/d \
+ "/<pre id=\"blob\">" d d i "$(lowdown file/README.md)" . w \
+| ed -s file/README.md.html > /dev/null
+```
+
+### stagit-index
+
+Just a quick mention to how I use stagit-index, the command used to
+generate the index page.
+The only change I make from the default configuration is to change
+the links to each repository to point to the file list instead
+of the log page. stagit-index writes its result to standard output, so
+I can simply use `sed`:
+
+```
+stagit-index /home/git/*.git | sed 's|/log\.html||g' > "$basedir/index.html"
+```
+
+And that's it. Well, I also copy the style files and add a bottom bar,
+and change the title from a `<span class="desc">` to an `<h1>` element,
+again using `ed`. If you want to see the details you can check them out
+[here](https://git.tronto.net/git-hooks/file/post-receive-stagit.html).
+
+## Conclusions
+
+stagit is the perfect minimalist tool to publish your git repository
+with a simple, static web interface. It requires nothing more than
+an http server capable of serving html files. Static files are also
+very simple to customize and tune to your needs.
+
+I have wanted to make this post for quite some time now, mainly
+as an excuse to clean up and document my scripts. I finally had some
+time to work on this - even if scattered around multiple days.
+
+As always, I have tried but failed to keep my post short - I am too
+eager to explain everything I know as clearly as possbile!
+I hope you enjoyed or found it useful. If you have questions or comments,
+feel free to send me an [email](mailto:sebastiano@tronto.net).
diff --git a/src/blog/blog.md b/src/blog/blog.md
@@ -2,6 +2,8 @@
[RSS Feed](feed.xml)
+* 2022-11-23 [Self-hosted git pages with stagit (featuring ed, the standard editor)](2022-11-23-git-host)
+* 2022-10-19 [Keeping my email sorted (the hard way)](2022-10-19-email-setup)
* 2022-10-01 [The man page reading club: tetris(6)](2022-10-01-tetris)
* 2022-09-20 [The man page reading club: sh(1) - part 2: commands and builtins](2022-09-20-sh-2)
* 2022-09-13 [The man page reading club: sh(1) - part 1: shell grammar](2022-09-13-sh-1)
diff --git a/src/blog/feed.xml b/src/blog/feed.xml
@@ -9,6 +9,20 @@ Thoughts about software, computers and whatever I feel like sharing
</description>
<item>
+<title>Self-hosted git pages with stagit (featuring ed, the standard editor)</title>
+<link>https://sebastiano.tronto.net/blog/2022-11-23-git-host</link>
+<description>Self-hosted git pages with stagit (featuring ed, the standard editor)</description>
+<pubDate>2022-11-23</pubDate>
+</item>
+
+<item>
+<title>Keeping my email sorted (the hard way)</title>
+<link>https://sebastiano.tronto.net/blog/2022-10-19-email-setup</link>
+<description>Keeping my email sorted (the hard way)</description>
+<pubDate>2022-10-19</pubDate>
+</item>
+
+<item>
<title>The man page reading club: tetris(6)</title>
<link>https://sebastiano.tronto.net/blog/2022-10-01-tetris</link>
<description>The man page reading club: tetris(6)</description>
