How this website is created

Published on 2024-04-30 (updated on 2024-09-28)

tags: emacs org

It’s been a while since I updated this site. Most of the old content was tailored for job applications. I have less reasons now to use this as a portfolio. Rather just a space to host my thoughts. Figured I will rebuild the site from scratch. I primarily write all my posts and notes in Org. Earlier I had successively used ox-hugo (+ Hugo) and blorg convert those to HTML. Nice and fast tools, but didn’t neatly fit into my workflow.

This time around I wanted to cut out all the middlemen and just use Emacs and Org to build this site. I have almost succeeded , but for a minor Python bit. There are a bazillion blogs and tutorials on using Emacs and Org to build a static site. I found this and this particularly useful. Crawling /r/emacs will give you dozens more. I have decided on the following to keep things relatively stable, and not reinvent the wheel:

This Emacs + Org + Python publishing system might be slower than some of the more popular static site generators. But in the absence of any benchmarks all I can say is that I really do not see any difference. On the other hand this system is self-contained, stable, and modular. My hope is that now that I have it set up, I will not have to tinker with it much.

Directory structure

The directory structure of this website is as follows (produced with M-x speedbar which does not show hidden files like .htaccess, etc.): 

0:<+> css
0:<+> images
0:<+> posts
0:<+> tags
0:[+] 404.html *
0:[+] 404.org *
0:[+] Makefile *
0:[+] index.html *
0:[+] index.org *
0:[+] publish.el *

Additional CSS files live in the css directory:

  1. extra.css adds some styling not handled by simplecss.
  2. syntax.css produced with htmlize-buffer from htmlize.el handles code highlighting.

Images used outside of posts, such as my profile picture, live in the images directory.

Posts

All posts and any associated files, such as images, live in their own directories under the posts directory. This setup of the posts makes it very easy to remove any post, and I do not need to worry about links to images and such. At the time of updating this post, the posts directory looks like as follows: 

0:<-> posts
1: <+> braketx
1: <-> how-this-website-is-created
2:  [+] index.html *
2:  [+] index.org *
1: [+] index.html *
1: [+] index.org *

Tags

The tags directory is similarly arranged as the posts directory, one sub-directory for each tag. Each sub-directory contains an index.org file which lists all the posts corresponding to that tag. Originally I was using Python to create this, but now I have figured out an Emacs Lisp solution. To be honest I feel that the Emacs Lisp solution is simpler.

My original idea was:

  1. Build a hash table with tags as keys and list of corresponding posts as values.
  2. Create sub-directories for each tag under tags.
  3. Write the contents of the hash table to the index.org files in the sub-directories under tags.
  4. Generate HTML files for the tags with Org publishing.
  5. Link each tag HTML file from its corresponding post HTML files.

The last bit required post-processing the generated HTML files. I failed to implement this with Emacs’s hash table. Implementing this with Python’s dict and regex searches was simple. But it was a fragile system. Changes, whether unintentional or due to carelessness, in the generated HTML can lead to spurious regex matches. My current Emacs Lisp solution is:

  1. Collect the tags for a post.
  2. For each tag, if tags/<tag>/index.org exists, then append relative link to the post to it, else create that file and add relative link to the post to it.
  3. Repeat this for all posts.

Linking tags from posts is handled by my header function which uses built-in Org variables. This bypasses both post-processing the generated HTML, and regex matches. When publishing, I just recursively delete the tags directory and recreate it. This gets around the issue of checking existing links to posts for duplicates.

Publishing code

All the Emacs Lisp code are in publish.el at the root of my pages directory. The main functions are:

my-html-preamble
This creates the <header> element for each page. It is based on this. It automatically adds the date that each post was published on. If the post was updated later then it also adds that date. Additionally it adds links to the tags for that post.
my-sitemap-format-entry
This function defines how each entry in the list of posts look like. It adds links to the tags for each post.
my-create-tag-file-for-tag
This function encodes the logic for step 2 of my new solution for tags. my-create-tag-files-from-post-file and my-create-tag-files-from-directory then loops this function over all tags and all posts.
;;; Publishing configuration -*- lexical-binding: t -*-
(require 'package)
(package-initialize)
(require 'org)
(require 'ox)
(require 'ox-publish)

(setq org-ditaa-jar-path "/usr/share/ditaa/ditaa.jar")
(org-babel-do-load-languages 'org-babel-load-languages
                             (append org-babel-load-languages
                                     '((ditaa . t))))

;;; header / footer
(defvar my-html-head-extra
  (concat
   "<link rel=\"icon\" href=\"data:image/svg+xml,<svg xmlns=%22http://www.w3.org/2000/svg%22 viewBox=%220 0 100 100%22><text y=%22.9em%22 font-size=%2290%22>স</text></svg>\">\n"
   "<link rel=\"stylesheet\" href=\"https://cdn.simplecss.org/simple.min.css\">\n"
   "<link rel=\"stylesheet\" type=\"text/css\" href=\"/css/syntax.css\">\n"
   "<link rel=\"stylesheet\" type=\"text/css\" href=\"/css/extra.css\">\n"))

(defun my-html-preamble (info)
  (let* ((date-format "%Y-%m-%d")
         (file (plist-get info :input-file))
         (published (org-export-data
                     (org-export-get-date info date-format) info))
         (updated (format-time-string
                   date-format
                   (file-attribute-modification-time
                    (file-attributes file))))
          (title (org-get-title info)))
      (concat
       "<nav>\n"
       "<a href=\"/\" alt=\"Go home\">home</a>\n"
       "<a href=\"/posts/\" alt=\"Read all posts\">posts</a>\n"
       "</nav>\n"
       "<h1 class=\"title\">" title "</h1>\n"
       (unless (or (equal title "~")
                   (equal title "~/posts")
                   (string-match-p (regexp-quote "tags")
                                   (directory-file-name file)))
         (concat "<p class=\"date\">Published on " published
                 (unless (equal updated published)
                   (concat " (updated on " updated ")"))
                 "</p>\n"))
       (when org-file-tags
         (concat "<p class=\"tags\">tags: "
                 (mapconcat
                  '(lambda (tag)
                     (concat "<a class=\"tag\" href=\"../../tags/"
                             tag
                             "/index.html\">"
                             tag
                             "</a>"))
                  (flatten-list org-file-tags)
                  " ")
                 "</p>\n")))))

(defvar my-html-postamble
   (format-time-string "<p>Copyright &copy; 2020 &mdash; %Y Soham Pal</p>\n"))

;;; Sitemap
(defun my-sitemap-format-entry (entry style project)
  (let ((title (org-publish-find-title entry project))
        (tags (org-publish-find-property entry :filetags project)))
    (concat "[[file:" entry "][" title "]]"
            (when tags
              (concat
               " ("
               (mapconcat
                '(lambda (tag)
                   (concat "[[file:../tags/" tag "/index.org][" tag "]]"))
                (flatten-list tags)
                " ")
               ")")))))

;;; Tags
(defun my-create-tag-file-for-tag (tag post-file)
  (let* ((tag-file (file-name-concat "tags" tag "index.org"))
         (relpath (file-relative-name post-file (file-name-directory tag-file)))
         (title (org-get-title post-file)))
    (if (file-exists-p tag-file)
        (write-region
         (concat "- [[file:" relpath "][" title "]]\n")
         nil
         tag-file
         :append)
      (progn
        (make-directory (file-name-directory tag-file) :parents)
        (with-temp-file tag-file
          (insert (concat "#+title:" tag "\n"))
          (insert "\n")
          (insert (concat "- [[file:" relpath "][" title "]]\n")))))))

(defun my-create-tag-files-from-post-file (post-file)
  (let ((tags (with-temp-buffer
                (insert-file-contents post-file)
                (org-mode)
                 org-file-tags)))
    (when tags
      (dolist (tag (flatten-list tags))
        (my-create-tag-file-for-tag tag post-file)))))

(defun my-create-tag-files-from-directory (dir)
  (let ((files (directory-files-recursively dir ".org")))
    (dolist (file files)
      (my-create-tag-files-from-post-file file))))


;;; Common publish settings
(setq org-export-allow-bind-keywords t
      org-export-babel-evaluate nil
      org-export-with-author nil
      org-export-with-creator nil
      org-export-with-footnotes t
      org-export-with-latex t
      org-export-with-smart-quotes t
      org-export-with-section-numbers nil
      org-export-with-toc nil
      org-export-with-title nil
      org-html-container-element "main"
      org-html-doctype "html5"
      org-html-divs '((preamble "header" "preamble")
                      (content "main" "content")
                      (postamble "footer" "postamble"))
      org-html-head-include-default-style nil
      org-html-head-extra my-html-head-extra
      org-html-html5-fancy t
      org-html-htmlize-output-type "css"
      org-html-link-home ""
      org-html-link-up ""
      org-html-preamble 'my-html-preamble
      org-html-postamble my-html-postamble
      org-html-validation-link nil)

;;; Export
(setq org-publish-project-alist
      `(("org-posts"
         :base-directory "./posts/"
         :base-extension "org"
         :publishing-directory "./posts"
         :recursive t
         :publishing-function org-html-publish-to-html
         :auto-sitemap t
         :sitemap-filename "index.org"
         :sitemap-style list
         :sitemap-format-entry my-sitemap-format-entry
         :sitemap-sort-files anti-chronologically
         :sitemap-title "~/posts")
        ("org-tags"
         :base-directory "./tags/"
         :base-extension "org"
         :publishing-directory "./tags"
         :recursive t
         :publishing-function org-html-publish-to-html)
        ("org-home"
         :base-directory "./"
         :base-extension "org"
         :publishing-directory "./"
         :recursive nil
         :publishing-function org-html-publish-to-html)))

;;; Publish
(when (file-exists-p "tags")
  (delete-directory "tags" :recursive nil))
(my-create-tag-files-from-directory "posts")
(org-publish-all)

Makefile

Finally I have a Makefile to semi-automate the whole process. It has directives to:

  1. Run publish.el with Emacs in batch mode.
  2. Delete the Org cache.
  3. Delete the Org cache, then run publish.el with Emacs in batch mode.

However usually I just find myself doing C-c C-e in a buffer visiting publish.el. 

build:
        emacs -Q --batch --script publish.el

clean:
        rm ~/.org-timestamps/org-*

clean_and_build: | clean build