sebastiano.tronto.net

Source files and build scripts for my personal website
git clone https://git.tronto.net/sebastiano.tronto.net
Download | Log | Files | Refs | README

website.md (10718B)


      1 # How I update my website
      2 
      3 When I created my website, I decided that I wanted to understand 100% of what
      4 I did. In practice, this means that I did not want to use any framework, not
      5 even a simple one. Someone might call this *minimalism*, someone else might
      6 call it *being a control freak*. I think I like the second one more.
      7 
      8 The principles I follow, which are actually an afterthought after a few months
      9 of trial-and-error, are roughly these:
     10 
     11 * **Minimalism**, a.k.a **control-freakism**: As mentioned above, I want to
     12   only use tools that I understand completely,
     13   at the cost of writing code by hand sometimes.
     14 * **Ease of use**: I want to write the bulk of my pages in
     15   [Markdown](https://en.wikipedia.org/wiki/Markdown), which is easier to
     16   read and edit than html code.
     17 * **Reproducibility**: Ideally, building new html pages (from a
     18   newly-written Markdown document) and uploading them to my server
     19   should be done by issuing simple commands such as `make` and `make deploy`.
     20 
     21 In more practical terms, this boils down to writing a couple of CSS and html
     22 files, writing a script that adds a header and a footer to the output of
     23 [lowdown](https://kristaps.bsd.lv/lowdown/), and using
     24 [rsync](https://en.wikipedia.org/wiki/Rsync) to deploy the files to my server.
     25 All of this is available on
     26 [my git page](https://git.tronto.net/sebastiano.tronto.net/), but I won't
     27 explain every detail of these build scripts here. In particular, my script
     28 also builds a [gemini](https://sebastiano.tronto.net/blog/2022-06-04-gemini/)
     29 version of my website, which I won't discuss here.
     30 
     31 ## Prerequisites
     32 
     33 My website is hosted on an OpenBSD virtual machine in a remote server. I can
     34 access this virtual machine via
     35 [SSH](https://en.wikipedia.org/wiki/Secure_Shell), which gives me root
     36 access to the operating system. I use rsync for uploading my files to this
     37 server. As long as you have an http server that can serve static html
     38 files and a way to upload your files to this server, you can easily manage
     39 your website in a similar way. I am not going to explain how to do all of this
     40 here; if you need help I suggest you have a look at
     41 [Roman Zolotarev's website](https://rgz.ee).
     42 
     43 As for my local machine, the one I am actually using to write these blog
     44 posts, I just use a Markdown translator, a text editor, rsync
     45 and other basic UNIX tools.
     46 
     47 ## Directory structure
     48 
     49 In my working directory
     50 there are two main folders with the exact same sub-folder structure: the first
     51 one is `src`, which contains all the markdown files I write, plus any other
     52 file I need for my pages, such as pictures; the other is `http`, which contains
     53 the html pages exactly as they are on my website. The `http` folder is
     54 generated from the src folder when I run the `build.sh` script - more
     55 on this later! (You won't find the `http` folder on my git page)
     56 
     57 There is one small caveat here:
     58 I like my urls to be clean and I want them stripped of the `.html`
     59 extension. To do this, I set up my `src` folder so that every
     60 subfolder contains at most one `.md` (Markdown) file, which is converted to
     61 an `index.html` file in the corresponding subfolder of `http`.
     62 In practice, if there are the following files:
     63 
     64 ```
     65 ├── src
     66 │   ├── git
     67 │   │   └── git-or-any-other-name.md
     68 ```
     69 
     70 The following is generated by `build.sh`:
     71 
     72 ```
     73 ├── http
     74 │   ├── git
     75 │   │   └── index.html
     76 ```
     77 
     78 So that when one accesses
     79 [sebastiano.tronto.net/git](https://sebastiano.tronto.net/git/)
     80 the web server automatically serves the `index.html` file.
     81 Without this trick the correct URL would have been
     82 `sebastiano.tronto.net/git.html` or something, which I don't like.
     83 
     84 The main working directory also contains the 
     85 [`top.html`](https://git.tronto.net/sebastiano.tronto.net/file/top%2Ehtml%2Ehtml)
     86 and
     87 [`bottom.html`](https://git.tronto.net/sebastiano.tronto.net/file/bottom%2Ehtml%2Ehtml)
     88 files. These files are not uploaded directly to my server - they are not
     89 even well-formed html files! - but they are used to build all other
     90 html pages.
     91 
     92 ## Building the pages
     93 
     94 The basic idea behind `build.sh` is very simple. If we want to create the
     95 html file corresponding to, say, `src/page/file.md`, we just need to create
     96 a file called `http/page/index.html` and copy there the contents of
     97 `top.html`, followed by the output of `lowdown src/page/file.md`, followed
     98 by the contents of `bottom.html`. In shell language:
     99 
    100 ```
    101 cat top.html > http/page/index.html
    102 lowdown src/page/file.md >> http/page/index.html
    103 cat bottom.html >> http/page/index.html
    104 ```
    105 
    106 Of course you don't have to use lowdown as I do: any Markdown translator
    107 works. Indeed, if you want to use a different markup language to write
    108 your pages you just need to replace the second line in the code above.
    109 
    110 We would also like to make a small change to the header while
    111 we build this page, namely we want the title of the page (the one
    112 displayed in your browser's top bar or tab) to match the actual
    113 title of the page or blog post. To do this easily I have left a
    114 placeholder `TITLE` in `top.html`, that we just need to replace with the
    115 actual title of the page. To find out what the title is we just need to
    116 get the text following
    117 the first `# ` (hash space) in the Markdown file - that is, the first
    118 "big title" of the page. We can do this
    119 thanks to the classic UNIX tools sed, grep and head:
    120 
    121 ```
    122 sed "s/TITLE/$(grep '^\# ' < src/page/file.md \
    123 	| head -n 1 | sed 's/^\# //')/" < top.html > http/page/index.html
    124 lowdown src/page/file.md >> http/page/index.html
    125 cat bottom.html >> http/page/index.html
    126 ```
    127 
    128 The first two lines might be a bit complicated to work out if you
    129 are not familiar with these commands. Let's break them down!
    130 
    131 The main command is `sed "s/TITLE/...stuff.../"` which replaces the
    132 first occurrence of the string `TITLE` with that complicated stuff.
    133 The end of the second line tells sed to use `top.html` as input and
    134 write the output to `http/page/index.html`. The complicated stuff
    135 that is going to replace `TITLE` is enclosed in `$()`, which means
    136 that it is the result of a command. This command is itself a chain
    137 of commands: first we find all lines that start with `# ` with
    138 `grep '^\# '` on the correct file (`< src/page/file.html`), then
    139 we take the first of these lines (`head -n 1`) and finally we
    140 trim the leading `# ` with `sed`. As you can see, the UNIX shell
    141 is quite a powerful tool!
    142 
    143 Now we just need to do all of this recursively on the `src` folder.
    144 The final result looks something like this:
    145 
    146 ```
    147 #!/bin/sh
    148 
    149 recursivebuild() {
    150 	local destdir=$(echo $1 | sed 's|^src|http|')
    151 	mkdir -p "$destdir"
    152 	for file in $(ls $1); do
    153 		if [ -d "$1/$file" ]; then
    154 
    155 			# Recursively build subdirectories
    156 			mkdir -p "$destdir/$file"
    157 			recursivebuild "$1/$file"
    158 		else
    159 			extension=$(echo "$file" | sed 's/.*\.//')
    160 			if [ "$extension" = "md" ]; then
    161 
    162 				# Process Markdown files, as above
    163 				sed "s/TITLE/$(grep '^\# ' < "$1/$file" \
    164 					| head -n 1 \
    165 					| sed 's/^\# //')/" < top.html \
    166 					> "$destdir/index.html"
    167 				lowdown "$1/$file" >> "$destdir/index.html"
    168 				cat bottom.html >> "$destdir/index.html"
    169 			else
    170 				
    171 				# Copy all other files as they are
    172 				cp "$1/$file" "$destdir/$file"
    173 			fi
    174 		fi
    175 	done
    176 }
    177 
    178 recursivebuild src
    179 ```
    180 
    181 ## Extras: the blog index and RSS feed
    182 
    183 The [blog index page](https://sebastiano.tronto.net/blog/) is also
    184 generated by the build script, but the corresponding Markdown file
    185 in `src` is not created by hand. Instead, this file is generated by
    186 scanning the `src/blog` subfolder. For each post, the date is deduced
    187 from the name of the folder containing the markdown file, which always
    188 starts with the date itself in the `yyyy-mm-dd` format.
    189 
    190 While we scan the blog directory to create a list of posts, we might as
    191 well make an [RSS feed](https://en.wikipedia.org/wiki/RSS) file for the
    192 blog. This is a file used by feed reader applications to check if there
    193 is any new post. The format is quite simple: check out
    194 [mine](https://sebastiano.tronto.net/blog/feed.xml).
    195 
    196 The code to accomplish this looks something like this:
    197 
    198 ```
    199 makeblog() {
    200 	bf=src/blog/blog.md    # Blog index file
    201 	ff=src/blog/feed.xml   # RSS feed file
    202 
    203 	printf "# Blog\n\n[RSS Feed](feed.xml)\n\n" > $bf
    204 	cp feed-top.xml $ff
    205 
    206 	for i in $(ls src/blog | sort -r); do
    207 		if [ -d src/blog/$i ]; then
    208 
    209 			# Get basic data of the post (date, title)
    210 			f="src/blog/$i/*.md"
    211 			d=$(echo $i | grep -oE '^[0-9]{4}-[0-9]{2}-[0-9]{2}')
    212 			t=$(head -n 1 $f | sed 's/# //')
    213 
    214 			# Add blog post to the list
    215 			echo "* $d [$t]($i)" >> $bf
    216 
    217 			# Create RSS feed item
    218 			echo "<item>" >> $ff
    219 			echo "<title>$t</title>" >> $ff
    220 			echo "<link>https://sebastiano.tronto.net/blog/$i</link>" >> $ff
    221 			echo "<description>$t</description>" >> $ff
    222 			echo "<pubDate>$d</pubDate>" >> $ff
    223 			echo "</item>" >> $ff
    224 			echo "" >> $ff
    225 		fi
    226 	done
    227 
    228 	# Close the RSS feed file
    229 	echo "" >> $ff
    230 	echo "</channel>" >> $ff
    231 	echo "</rss>" >> $ff
    232 }
    233 ```
    234 
    235 ## Deploying with make
    236 
    237 Updating or adding a page is now very easy: I just need to edit the
    238 corresponding Markdown file, run `./build.sh` to build the new html
    239 pages and run
    240 
    241 ```
    242 rsync -rv --delete --rsync-path=openrsync http/ \
    243 	tronto.net:/var/www/htdocs/sebastiano.tronto.net
    244 ```
    245 
    246 to sync the `http` directory with my server. I need to use the
    247 `--rsync-path` option because the `rsync` binary has a different 
    248 name on my local system (Linux) than on my server (OpenBSD). But apart from
    249 this the command is straightforward.
    250 
    251 Of course I don't want to type this lenghty command every time. It is very
    252 convenient in this case to write a short Makefile:
    253 
    254 ```
    255 all: clean
    256 	./build.sh
    257 
    258 clean:
    259 	rm -r http
    260 	mkdir -p http
    261 
    262 deploy:
    263 	rsync -rv --delete --rsync-path=openrsync http/ \
    264 		tronto.net:/var/www/htdocs/sebastiano.tronto.net
    265 
    266 .PHONY: all clean deploy
    267 ```
    268 
    269 So that I just need to run `make` to build and `make deploy` to upload the
    270 new files. Watch out: if you want to reproduce this on your system, make
    271 sure that the user on your server has sufficient permissions to run
    272 that rsync command - in particular you need write permission on the
    273 `/var/www/htdocs` folder.
    274 
    275 If you are not familiar with the [make(1)](https://man.openbsd.org/make)
    276 syntax, this step is completely optional and you can simply type
    277 the full commands every time, or make another small script called
    278 `deploy.sh` and run that instead.
    279 
    280 ## Follow-up?
    281 
    282 I am sure my build scripts will keep evolving over time, so at some point I 
    283 might write a new post about the same topic. I am also probably going to write
    284 something about how I generate my [git page](https://git.tronto.net/) using
    285 [stagit](https://codemadness.org/stagit.html), if anything just to document
    286 my post-receive hooks. So, if you liked this post, stay tuned for more!