Nicolas Martyanoff – Brain dump

Working with Common Lisp pathnames

nicolas@n16f.net (Nicolas Martyanoff) — Sat, 26 Apr 2025 18:00:00 +0000

Common Lisp pathnames, used to represent file paths, have the reputation of being hard to work with. This article aims to change this unfair reputation while highlighting the occasional quirks along the way.

Filenames and file paths

The distinction between filename and file paths is not always obvious. On POSIX systems, the filename is the name of the file, while a file path represents its absolute or relative location in the file system. Which also means that all filenames are file paths, but not the other way around.

Common Lisp uses the term filename for objects which are either pathnames or namestrings, both being representation of file paths. We will try to avoid confusion by using the terms filenames, pathnames and namestrings when referring to Common Lisp concepts and we will talk about file paths when referring to the language-agnostic file system concept.

Pathnames

Pathnames are an implementation-independent representation of file paths made of six components:

an host identifying either the file system or a logical host;
a device identifying the logical of physical device containing the file;
a directory representing an absolute or relative list of directory names;
a name;
a type, a value nowadays known as file extension;
a version, because yes file systems used to support file versioning.

While this representation might seem way too complicated —it originates from a time where the file system ecosystem was much richer— it still is suitable for modern file systems.

The make-pathname function is used to create pathnames and lets you specificy all components. For example the following call yields a pathname representing the file path represented on POSIX systems by /var/run/example.pid:

(make-pathname :directory '(:absolute "var" "run") :name "example" :type "pid")

Common Lisp functions manipulating file paths of course accept pathnames, letting you keep the same convenient structured representation everywhere, only converting from/to a native representation at the boundaries of your program.

Special characters

What happens when you construct a pathname with components containing separation characters, e.g. a directory name containing / on a POSIX system or a type containing .? According to Common Lisp 19.2.2.1.1, the behaviour is implementation-defined; but if the implementation accepts these component values it must handle quoting correctly.

For example:

CLISP rejects separator characters in component values, signaling a SIMPLE-ERROR condition.
CCL accepts them and quotes them when converting the pathname to a namestring. So (namestring (make-pathname :name "foo/bar" :type "a.b")) yields "foo\\/bar.a\\.b" on Linux.
SBCL accepts and quotes them but does not quote . in type components, yielding "foo\\/bar.a.b" for the example above.
ECL accepts them but fails to quote them when converting the pathname to a namestring.

One could wonder about which implementation, CCL or SBCL, is correct regarding the quoting of the . character in type strings on POSIX platforms. While everyone understands that / is special in file and directory names, . is debatable because POSIX does not mention the type extension in its definitions: foo.txt is the name of the file, not a combination of a name and a type. As such, I would argue that quoting and not quoting are both correct. And as you will realize then reading about namestrings further in this article, it is irrelevant since namestrings are not POSIX paths.

Note that whether ECL violates the standard or not is unclear since there is no character quoting for POSIX paths. In other words, there is no such thing as a directory named a/b, because it could not be referenced in a way different from a directory named b in a directory named a. This behaviour derives directly from POSIX systems treating paths as strings and not as structured objects.

Invalid characters

The Common Lisp standard mentions special characters but is silent on the subject of invalid characters. For example POSIX forbids null bytes in filenames. But since it is not a separation character, implementations are free to deal with it as they see fit.

When testing implementations with a pathname containing a null byte using (make-pathname :name (string (code-char 0))), CCL, SBCL and ECL accept it while CLISP signals an error mentioning an illegal argument.

I am not convinced by CLISP’s behaviour since null bytes are only invalid in POSIX paths, not in Common Lisp filenames, meaning that the error should occur when the pathname is internally converted to a format usable by the operating system.

Pathname component case

A rarely mentioned property of pathnames is the support for case conversion. MAKE-PATHNAME and function returning pathname components (e.g. PATHNAME-TYPE) support a :CASE argument, either :COMMON or :LOCAL indicating how how to handle character case in strings.

With :LOCAL —which is the default value—, these functions assume that component strings are already represented following the conventions of the underlying operating system. It also dictates that if the host only supports one character case, strings must be returned converted to this case.

With :COMMON, these functions will use the default (customary) case of the host if the string is provided all uppercase, and the opposite case if the string is provided all lowercase. Mixed case strings are not transformed.

These behaviours are not intuitive and made much more sense at a time where some file systems only supported one specific case. You should probably stay away from component case handling unless you really know what you are doing.

On a personal note, as someone running Linux and FreeBSD, I am curious about the behaviour of various implementations on Windows and MacOS since both NTFS and APFS are case insensitive.

Unspecific components

While all components can be null, some of them can be :UNSPECIFIC (which ones is implementation-defined). The only real use case for :UNSPECIFIC is to affect the behaviour of MERGE-PATHNAMES: if a component is null, the function uses the value of the component in the pathname passed as the :DEFAULTS argument; if a component is :UNSPECIFIC, the function uses the same value in the resulting pathname.

For example:

(merge-pathnames (make-pathname :name "foo")
                 (make-pathname :type "txt"))

yields the "foo.txt" namestring, but

(merge-pathnames (make-pathname :name "foo" :type :unspecific)
                 (make-pathname :type "txt"))

yields "foo".

Unfortunately the inability to rely on its support for specific component types (since it is implementation-defined) makes it interesting more than useful.

Namestrings

Namestrings are another represention for file paths. While pathnames are structured objects, namestrings are just strings. The most important aspect of namestrings is that unless they are logical namestrings (something we will cover later), the way they represent paths is implementation-defined (c.f. Common Lisp 19.1.1 Namestrings as Filenames). In other words the namestring for the file foo of type txt in directory data could be data/foo.txt. Or data\foo.txt. Or data|foo#txt. Or any other non-sensical representation. Fortunately implementations tend to act rationally and use a representation as similar as possible to the one of their host operating system.

One should always remember that even though namestrings look and feel like paths, they are still a representation of a Common Lisp pathname, meaning that they may or may not map to a valid native path. The most obvious example would be a pathname whose name is the null byte, created with (make-pathname :name (string (code-char 0))), whose namestring is a one character string that has no valid native representation on modern operating systems.

Pathnames can be converted to namestrings using the NAMESTRING function, while namestrings can be parsed into pathnames with PARSE-NAMESTRING. The #P reader macro uses PARSE-NAMESTRING to read a pathname. As such, #P"/tmp/foo.txt" is identical to #.(parse-namestring '"/tmp/foo.txt").

Note that most functions dealing with files will accept a pathname designator, i.e. either a pathname, a namestring or a stream.

Native namestrings

An unfortunately missing feature from Common Lisp is the ability to parse native namestrings, i.e. paths that use the representation of the underlying operating system.

To understand why it is a problem, let us take *.txt, a perfectly valid filename at least on any POSIX platform. In Common Lisp, you can construct a pathname representing this filename with (make-pathname :name "*" :type "txt"). No problem whatsoever. However the "*.txt" namestring actually represents a pathname whose name component is :WILD. There is no namestring that will return this pathname when passed to PARSE-NAMESTRING.

As a result, when processing filenames coming from the external world (a command line argument, a list of paths in a document, etc.), you have no way to handle those that contain characters used by Common Lisp for wild components.

There is no standard way of solving this issue. Some implementations provide functions to parse native namestrings, e.g. SBCL with SB-EXT:PARSE-NATIVE-NAMESTRING or CCL with CCL:NATIVE-TO-PATHNAME. If you use ASDF, you can also use UIOP:PARSE-NATIVE-NAMESTRING.

Wildcards

Up to now pathnames may have looked like a slightly unusual representation for paths. But we are just getting started.

Pathname can be wild, meaning that they contain one or more wild components. Wild components can match any value. All components can be made wild with the special value :WILD. Directory elements also support :WILD-INFERIORS which matches one or more directory levels.

As such

(make-pathname :directory '(:absolute "tmp" :wild) :name "foo" :type :wild)

is equivalent to the /tmp/*/foo.* POSIX glob pattern, while

(make-pathname :directory '(:absolute "tmp" :wild-inferiors "data" :wild)
               :name :wild :type :wild)

is equivalent to /tmp/**/data/*/*.*.

Wild pathnames only really make sense for the DIRECTORY function which returns files matching a specific pathname.

Logical pathnames

We currently have talked about pathnames representing either paths to physical files or pattern of filenames. Logical pathnames go further and let you work with files in a location-independent way.

Logical pathnames are based on logical hosts, set as pathname host components. Logical pathnames can be passed around and manipulated as any other pathnames; when used to access files, they are translated to a physical pathname, i.e. a pathname referring to an actual file in the file system.

SBCL uses logical pathnames for source file locations. While SBCL is shipped with its source files, their actual location on disk depends on how the software was installed. Instead of manually merging pathnames with a base directory value everywhere, SBCL uses the SYS logical host to map all pathnames whose directory starts with SRC to the actual location on disk. For example on my machine:

(translate-logical-pathname "SYS:SRC;ASSEMBLY;MASTER.LISP")

yields #P"/usr/share/sbcl-source/src/assembly/master.lisp".

Another example would be CCL which maps pathnames with the HOME logical host to subpaths of the home directory of the user.

Note that logical hosts are global to the Common Lisp environment. While SYS is reserved for the implementation, all other hosts are free to use by anyone. To avoid collisions, it is a good idea to name hosts after their program or library.

Logical namestrings

Logical namestrings are implementation-independent, meaning that you can safely use them in your programs without wondering about how they will be interpreted. Their syntax, detailed in section 19.3.1 of the Common Lisp standard, is quite different from usual POSIX paths. The host is separated from the rest of the path by a colon character, and directory names are separated by semicolon characters.

For example "SOURCE:SERVER;LISTENER.LISP" is the logical namestring equivalent of the /server/listener.lisp POSIX path for the SOURCE logical host.

The astute reader will notice the use of uppercase characters in logical namestrings. It happens that the different parts of a logical namestring are defined as using uppercase characters, but that the implementation translates lowercase letters to uppercase letters when parsing the namestrings (c.f. Common Lisp 19.3.1.1.7). We use the canonical uppercase representation for clarity.

Translation

Translation is controlled by a table that maps logical hosts to a list of pattern (wild pathnames or namestrings) and their associated wild physical pathnames.

One can obtain the list of translations for a logical host with LOGICAL-PATHNAME-TRANSLATIONS and update it with (SETF LOGICAL-PATHNAME-TRANSLATIONS). Each translation is a list where the first element is a logical pathname or namestring (usually a wild pathname) and the second element is a pathname or namestring to translate into.

The translation process looks for the first entry that satisfies PATHNAME-MATCH-P, which is guaranteed to behave in a way consistent with DIRECTORY. When there is match, the translation processes replaces corresponding patterns for each components.

And of course if translation results in a logical pathname, it will be recursively translated until a physical pathname is obtained.

A simple example would be the use of a logical host referring to a temporary directory. This lets a program manipulates temporary pathnames without having to know their actual physical location, the translation process being controlled in a single location.

(setf (logical-pathname-translations "tmp")
      (list (list "TMP:**;*.*.*"
                  (make-pathname :directory '(:absolute "var" "tmp" :wild-inferiors)
                                 :name :wild :type :wild))))

or if we were to use namestrings:

(setf (logical-pathname-translations "tmp")
      '(("TMP:**;*.*.*" "/var/tmp/**/*.*")))

Translating pathnames or namestrings using the TMP logical host yields the expected results. For example (translate-logical-pathname #P"TMP:CACHE;DATA.TXT") yields #P"/var/tmp/cache/data.txt".

Caveats

While logical pathnames are an elegant abstraction, they are plagued by multiple issues that make them hard to use correctly and in a portable way.

Logical namestring components can only contain letters, digits and hyphens (or the * and ** sequences for wild namestrings). This limitation probably comes from a need to be compatible with all existing file systems, but it can be a showstopper if one needs to refer to files whose naming scheme is not controlled by the program.

Namestring parsing is confusing: calling PARSE-NAMESTRING on an invalid namestring (because it contains invalid characters or because the host is not a known logical host) will not fail. Instead the string will be parsed as a physical namestring, introducing silent bugs. The LOGICAL-PATHNAME can be used to validate logical pathnames and namestrings.

The way translation converts between both pathname patterns is unclear. It is not specified by the Common Lisp standard. Debugging patterns can quickly become very frustrating, especially with implementations unable to produce quality error diagnostics.

Finally, the behaviour of logical pathnames with other functions is rarely obvious, leading to frustrating debugging sessions.

They nevertheless are a unique and helpful feature for very specific use cases.

Recipes

Resolving a path

Files are accessible through multiple paths. For example, on POSIX systems, foo/bar/baz.txt, foo/bar/../bar/baz.txt refer to the same file. If your operating system and file system support symbolic links, you can refer to the same physical file from multiple links, themselves being files.

It is sometimes useful to obtain the canonical path of a file. On POSIX systems, the realpath function serves this purpose. In Common Lisp, this canonical path is called truename, and the TRUENAME function returns it.

Transforming paths

The :DEFAULTS option of MAKE-PATHNAME is useful to construct a pathname that is a variation of another pathname. When a component passed to MAKE-PATHNAME is null, the value is taken from the pathname passed with :DEFAULTS.

For example to create the pathname of a file in the same directory as another pathname:

(make-pathname :name "bar"
               :defaults (make-pathname :directory '(:absolute "tmp") :name "foo"))

Or to create a wild pathname matching the same file names but with any extension:

(make-pathname :type :wild
               :defaults (make-pathname :name "foo" :type "txt"))

Or to obtain a pathname for the directory of a file:

(make-pathname :name nil
               :defaults (make-pathname :directory '(:relative "a" "b" "c")
                                        :name "foo"))

Joining two paths

Joining (or concatenating) two paths can be done with MERGE-PATHNAMES. In general calling (MERGE-PATHNAMES PATH1 PATH2) returns a new pathname whose components are taken either from PATH1 when they are not null, or from PATH2 when they are. As a special case, if the directory component of PATH1 is relative, the directory component of the result pathname is the concatenation of the directory components of both paths.

In other words

(merge-pathnames (make-pathname :directory '(:relative "x" "y"))
                 (make-pathname :directory '(:absolute "a" "b" "c")))

yields "/a/b/c/x/y/" but

(merge-pathnames (make-pathname :directory '(:absolute "x" "y"))
                 (make-pathname :directory '(:absolute "a" "b" "c")))

yields "/x/y/".

Finding files

The DIRECTORY function returns files matching a pathname, wild or not.

If the pathname is not wild, DIRECTORY returns a list of one or zero element depending on whether a file exists at this location or not.

If the pathname is wild, DIRECTORY behaves similarly to POSIX globs. Due to the way pathnames are structured, with the name and type being two different components, a common error is to specify a wild name without a type. In this case, DIRECTORY will not return any file with an extension (since their pathname has a non-null type). To match all files with any extension, set both the name and the type to :WILD.

Another interesting possibility is to only match directories. Directories are represented by pathnames with a non-null directory component and a null name component. Therefore to find all directories in /tmp (top-level only):

(directory (make-pathname :directory '(:absolute "tmp" :wild)))

Note that DIRECTORY returns truenames, i.e. pathnames representing the canonical location of the files. An unexpected consequence is that the function will resolve symlinks. Since the Common Lisp standard explicitely allows extra optional arguments, some implementations have a way to disable symlink resolving, e.g. SBCL with :RESOLVE-SYMLINKS or CCL with :FOLLOW-LINKS.

Resolving tildes in paths

It is commonly believed that tilde characters in paths is a universal feature. It is not. Tilde prefixes are defined in POSIX in the context of the shell (cf. POSIX 2017 2.6.1 Tilde Expansion) and are only supported in very specific locations.

To obtain the path of a file relative to the home directory of the current user, use the USER-HOMEDIR-PATHNAME function.

For example:

(merge-pathnames (make-pathname :directory '(:relative ".emacs.d")
                                :name "init" :type "el")
                 (user-homedir-pathname))

Controlling link opening in Emacs

nicolas@n16f.net (Nicolas Martyanoff) — Sun, 11 Aug 2024 18:00:00 +0000

Multiple Emacs modes recognize URIs so that you can follow them in a web browser, usually with a mouse left click. In modes that do not support it, one can still call browse-url-at-point interactively with the cursor positioned on a URI.

I wanted to make link handling more convenient for some time; this is what I ended up with.

Key bindings

There is no global Emacs concept of following links, so modes deal with them differently. I wanted to use the same key binding everywhere; I went for C-o (“o” for “open”) since I never use open-line. I also wanted the ability to use Eww —the Emacs builtin web browser— when it is convenient. So I added a function to select the secondary browser when the prefix is set. This way C-o opens Firefox, and C-u C-o opens Eww.

(setq browse-url-secondary-browser-function 'eww-browse-url)

(defun g-browse-url-at-point (arg)
  (interactive "P")
  (let ((browse-url-browser-function
         (if arg
             browse-url-secondary-browser-function
           browse-url-browser-function)))
    (browse-url-at-point)))

(keymap-global-set "C-o" 'g-browse-url-at-point)

Ultimately the function calls browse-url-at-point which identifies the URI under the cursor (i.e. at the current point) and calls browse-url.

Selecting a web browser

The browse-url function is fully configurable; two mechanisms are used to select a web browser for each link:

the browse-url-handlers and browse-url-default-handlers association lists, used to match specific URIs using predicates or regular expression;
the browse-url-browser-function variable (by default browser-url-default-browser) referencing a function which looks for available web browsers on the current machine.

Since I use multiple Firefox profiles, I needed the ability to open URIs in the right profile depending of the context. For example links in a professional email read in Gnus should be open in my work context.

The following function returns a handler that can be used in browse-url-handlers or as value for browse-url-browser-function:

(defun g-browse-url-firefox-function (profile)
  (lambda (uri &rest args)
    (let ((process-environment (browse-url-process-environment))
          (process-name (concat "firefox " uri))
          (process-args `("--new-tab" ,uri
                          ,@(if profile (list "-P" profile)))))
      (apply #'start-process process-name nil "firefox" process-args))))

For example, (g-browse-url-firefox-function "test") returns a browse function that will open the link in a new tab of a Firefox window associated with the “test” profile.

To change the logic deciding how to handle links in a buffer with a specific major mode, all you have to do is to add a hook to set browse-url-browser-function in a way that only affects the current buffer. Since this variable is not buffer-local by default, you have to take care to use setq-local when changing it.

Links in Gnus messages

Gnus messages are displayed using the gnus-article-mode major mode.

We first write a function to identify the account associated with the current group (assuming you are using nnimap for your email accounts as I do):

(defun g-current-gnus-account ()
  (let ((group gnus-newsgroup-name))
    (when (string-match "^nnimap\\+\\([^:]+\\):" group)
      (match-string 1 group))))

Then we define the actual browse function:

(defun g-gnus-browse-url (url &rest args)
  (let ((profile
         (pcase (g-current-gnus-account)
           ("my-company"
            "my-company-profile")
           ("a-client"
            "the-client-profile")
           (_
            "default"))))
    (apply (g-browse-url-firefox-function profile) url args)))

And the hook function:

(defun g-gnus-set-browser-function ()
  (setq-local browse-url-browser-function 'g-gnus-browse-url))

Finally we update the hook:

(use-package gnus-art
  :hook
  ((gnus-article-mode-hook . g-gnus-set-browser-function)))

Links in emails are now open in the right Firefox profile!

Links in the Circe IRC client

Same idea for Circe, select the right Firefox profile based on the network and channel of the current channel buffer.

We define the function to be called during the initialization of each channel buffer:

(defun g-circe-set-browser-function ()
  (let* ((network (with-circe-server-buffer
                    circe-network))
         (channel circe-chat-target)
         (profile (cond
                   ((and (equal network "my-network")
                         (equal channel "#my-channel"))
                    "the-right-profile")
                   (t
                    "default"))))
    (setq-local browse-url-browser-function
                (g-browse-url-firefox-function profile))))

And add it as a hook:

(use-package circe
  :hook
  ((circe-channel-mode-hook . g-circe-set-browser-function))

Taking control of your Go module paths

nicolas@n16f.net (Nicolas Martyanoff) — Sun, 07 Jul 2024 18:00:00 +0000

The canonical name of a Go module is its path. For third party modules, that means an URI. It is no secret that GitHub is currently the dominant code hosting platform, hence the fact that most Go modules are named github.com/<account>/<repository>. While I currently use GitHub for most of my repositories, I also want the ability to move out of it in the future. Fortunately the Go module system allows package authors to control how the package is accessed. One can use it to provide Go modules with their own domain and total control on module paths. Let us see how to do that.

How packages are downloaded

When the Go client downloads a module (either directly with GOPROXY=direct or when a proxy does it), it fetches the module URI with an HTTP request and a go-get=1 query parameter. The server includes a <meta> HTML element named go-import in the response, its value indicating how to actually download the module.

Let us check ourselves with my go-uuid module:

$ curl -s https://github.com/galdor/go-uuid\?go-get\=1 | grep go-import
  <meta name="go-import" content="github.com/galdor/go-uuid git https://github.com/galdor/go-uuid.git">

GitHub indicates that to download the github.com/galdor/go-uuid module, the client must clone the Git repository at https://github.com/galdor/go-uuid.git.

So for a module to be named example.com/foo, all you need is an HTTP server responding to requests at https://example.com/foo?go-get=1.

Using GitHub Pages

I wanted all my packages to be available at go.n16f.net/<name>. It is short, it looks good, and it opens the possibility to move the repositories out of GitHub in the future.

While I could have used my own HTTP server, I was worried about what would happen if someone needed to download a module when the server is down. Since I already use GitHub to host most repositories, I decided to use GitHub Pages: after all, what I need is a way to host HTML pages with the right <meta> tag at the right URI.

Generating repository pages

I created a GitHub repository for these pages. It was tempting to use a simple shell script to generate pages but I really did not want to use M4 and Ruby lets me use ERB templates, so Ruby it is.

All my Go modules are listed in a text file, and a small Ruby script generates both an index (not required but why not?) and a page for each module. The <meta> tag of each module page indicates that go.n16f.net/<name> can be accessed by cloning the Git repository at https://github.com/galdor/go-<name>. It is that simple.

Configuring the domain

Using your own domain means configuring it to point to GitHub pages.

First head to your domain name registrar website and configure your domain to point to GitHub pages. In my case, that means creating a CNAME DNS record named go.n16f.net. pointing to galdor.github.io.. You will notice that the GitHub Pages address is global to your account and not specific to each repository: GitHub takes care of routing, based on which domain is associated with each repository.

Then go to the GitHub Pages settings page and add your domain. You will have to create a TXT DNS record with a specific name and value to prove ownership of the domain.

Once done, go to the GitHub Pages settings page of the repository. Select the “GitHub Actions” source, and add the domain to “Custom domain”. Make sure to tick “Enforce HTTPS” while you are here.

Publishing pages

To generate and publish pages, we use a small GitHub Actions workflow. When new commits are pushed to the master branch, we run the script (the ubuntu-latest image already contains Ruby, so there is nothing to setup) and use official GitHub actions to upload and deploy the pages we just generated.

A quick check shows that the new URIs work as expected:

$ curl -s https://go.n16f.net/uuid\?go-get\=1 | grep go-import
    <meta name="go-import" content="go.n16f.net/uuid git https://github.com/galdor/go-uuid.git">

Updating modules

You can now update your Go modules: update the path in the go.mod file, and all import paths in source files. Yes this is annoying, and I would love a way to declare the path of each dependencies at project level, but Sed makes the task trivial.

Updating dependencies is a bit trickier. If your project depends on github.com/example/foo which you just renamed to example.com/foo, update all module paths in source files then run:

GOPROXY=direct go get example.com/foo@latest && go mod tidy

This will force Go to fetch the module directly from its source and not from the global proxy which may not be up-to-date yet.

Result

For a surprisingly small amount of work, I now have all my Go modules available at a path I fully control and I can add new modules by modifying a text file and letting the GitHub workflow handles the rest. And while the current setup is still dependent on GitHub, I can move it somewhere else without any change to module paths.

What is your excuse for not doing the same thing?

Pagination for database objects

nicolas@n16f.net (Nicolas Martyanoff) — Sat, 08 Jun 2024 18:00:00 +0000

So you have a server storing objects in a relational database, and an API, nowadays probably HTTP but it does not matter. Clients can fetch objects using the API. Obviously you do not want them loading too many objects at the same time, that would cause all kinds of performance issues. So you need a way to let clients request a manageable subset of all objects and iterate to finally obtain all objects. This is pagination.

I used to think it was a trivial problem. But given the number of complex and inefficient solutions I have seen over the years, I was clearly mistaken. Let us find out how to do it easily and efficiently.

Limit and offset

One of the first things you learn with SQL is how to use LIMIT and OFFSET to restrict the number of rows to retrieve.

Let us define a user table and load 10 users after the first 20:

CREATE TABLE users
  (id SERIAL PRIMARY KEY,
   name VARCHAR NOT NULL);

SELECT * FROM users
  LIMIT 10 OFFSET 20;

Have your clients send a limit and offset with each query, increment the offset to iterate. Easy, but there is a problem: the highest the offset is, the slower the request will be. LIMIT and OFFSET are applied after filtering, so if you have a million users, selecting the last 10 ones will still require processing the 999'990 before them. Not great.

Worse, this method can and will provide incorrect results when retrieving pages one after the other: adding rows to the table between each selection will make subsequent pages either miss rows or include some which we already returned in previous pages.

SQL Cursors

It can be tempting to use SQL cursors. After all, they let you consume rows iteratively without loading the whole result set in memory. But they are really not designed for the task: you would need to keep the associated database connection open between requests. Next idea.

Key-based pagination

Sometimes called “keyset” pagination, this method uses a key from the last row of the page you just loaded to obtain the next one efficiently.

Let us load the first page:

SELECT * FROM users
  ORDER BY id
  LIMIT 10;

Then for the next page, have the client provide the identifier of the last user in the page —let us say 42— and use it to load the second page:

SELECT * FROM users
  ORDER BY id
  WHERE id > 42
  LIMIT 10;

Continue the same way for subsequent pages.

As long as you are filtering on an indexed expression, the database will efficiently retrieve the rows for the page you requested. And you can use the exact same method to order results in the other direction, just invert the WHERE expression.

You will also avoid the inconsistencies of the LIMIT/OFFSET method: if the table is modified between two page retrieval operations, you may still miss rows (nothing you can do about it unless you are willing to retrieve the entire table in one operation), but you will not see rows from previous pages reappear in subsequent pages. Your users will thank you (or more accurately they will open less bug reports; even better).

Sorting

So you started using key-based pagination and everything looks fine, but now you need to sort results based on something else than the primary key. Your first idea could be to create an index on the column you want to use to sort, but it will not help: if multiple users have the same name, how will you select your pages?

We could use an index on two columns, name and id to obtain the strict total order we need then filter on these two columns:

CREATE INDEX users_name_id_idx
  ON users (name, id);

SELECT * FROM users
  WHERE (name, id) > ('bob', 42)
  ORDER BY name, id
  LIMIT 10;

But this is not standard SQL and would only work with databases supporting row-value comparisons.

Fortunately SQL lets you create indices on expressions and not just columns. So we create our own single-value keys by concatenating values. Of course we need a separator to avoid collisions: without it ('Bob', 42) and ('Bob4', 2) would both yield the same 'Bob42' key). The ASCII character set includes several separator characters which are perfect for this use case since they are not supposed to end up in your data (if they do, use another character); let us use the unit separator 0x1f:

CREATE INDEX users_name_pagination_idx
  ON users ((name || E'\x1f' || id));

Load the first page:

SELECT * FROM users
  ORDER BY name || E'\x1f' || id
  LIMIT 10;

And assuming the last user of the first page had the name Bob and the identifier 42, Load the second page.

SELECT * FROM users
  WHERE name || E'\x1f' || id > E'Bob\x1f42'
  ORDER BY name || E'\x1f' || id
  LIMIT 10;

Annoying to type but efficient and easy to generate. Of course you will want to make it easy for your clients: when returning a page, compute the key for the previous and next page and return them along the objects. The client can then trivially fetch the previous or next page.

Need to support multiple sort criteria? Just create an index for each of them.

That wasn’t so hard right?

Interactive Common Lisp development

nicolas@n16f.net (Nicolas Martyanoff) — Sun, 19 Nov 2023 18:00:00 +0000

Common Lisp programming is often presented as “interactive”. In most languages, modifications to your program are applied by recompiling it and restarting it. In contrast, Common Lisp lets you incrementally modify your program while it is running.

While this approach is convenient, especially for exploratory programming, it also means that the state of your program during execution does not always reflect the source code. You do not just define new constructs: you look them up, inspect them, modify them or delete them. I had to learn a lot of subtleties the hard way. This article is a compendium of information related to the interactive nature of Common Lisp.

Variables

In Common Lisp variables are identified by symbols. Evaluating (SETQ A 42) creates or updates a variable with the integer 42 as value, and associates it to the A symbol. After the call to SETQ, (BOUNDP 'A) will return T and (SYMBOL-VALUE 'A) will return 42.

You do not delete a variable: instead, you remove the association between the symbol and the variable. You do so with MAKUNBOUND. Following the previous example, (MAKUNBOUND 'A) will remove the association between the A symbol and the variable. And (BOUNDP 'A) returns NIL as expected. As for (SYMBOL-VALUE 'A), it now signals an UNBOUND-VARIABLE error as mandated by the standard.

What about DEFVAR and DEFPARAMETER? They are also used to declare variables (globally defined ones), associating them with symbols. Both define “special” variables (i.e. variables for which all bindings are dynamic; see CLtL2 9.2). The difference is that the initial value passed to DEFVAR is not evaluated if it already has a value. MAKUNBOUND will work on variables declared with DEFVAR or DEFPARAMETER as expected.

DEFCONSTANT is a bit more complicated. CLtL2¹ 5.3.2 states that “once a name has been declared by defconstant to be constant, any further assignment to or binding of that special variable is an error”, but does not clearly define whether MAKUNBOUND should or should not be able to be used on constants. However, CLtL2 5.3.2 also states that “defconstant […] does assert that the value of the variable name is fixed and does license the compiler to build assumptions about the value into programs being compiled”. If the compiler is allowed to rely on the value associated with the variable name, it would make sense not to allow the deletion of the binding. Thus it is recommended to only use constants for values that are guaranteed to never change, e.g. mathematical constants. Most of the time you want DEFPARAMETER.

Note that MAKUNBOUND does not apply to lexical variables.

Functions

Common Lisp is a Lisp-2, meaning that variables and functions are part of two separate namespaces. Despite this clear separation, functions behave similarly to variables.

Using DEFUN will either create or update the global function associated with a symbol. SYMBOL-FUNCTION returns the globally defined function associated with a symbol, and FMAKUNBOUND deletes this association.

Let us point out a common mistake when referencing functions: (QUOTE F) (abbreviated as 'F) yields a symbol while (FUNCTION F) (abbreviated as #'F) yields a function. The function argument of FUNCALL and APPLY can be either a symbol or a function (see CLtL2 7.3) It has two consequences:

First, one can write a function referencing F as (QUOTE F) with the expectation that F will later be bound to a function. The following function definition is perfectly valid even though F has not been defined yet:

(defun foo (a b)
  (funcall 'f a b))

Second, redefining the F function will update its association (or binding) to the F symbol, but the previous function will still be available if it has been referenced somewhere before the update. For example:

(setf (symbol-function 'foo) #'1+)
(let ((old-foo #'foo))
  (setf (symbol-function 'foo) #'1-)
  (funcall old-foo 42))

What about macros? Since macros are a specific kind of functions (CLtL2 5.1.4 “a macro is essentially a function from forms to forms”), it is not surprising that they share the same namespace and can be manipulated in the same way as functions with FBOUNDP, SYMBOL-FUNCTION and FMAKUNBOUND.

Symbols and packages

While functions and variables are familiar concepts to developers, Common Lisp symbols and packages are a bit more peculiar.

A symbol is interned when it is part of a package. The most explicit way to create an interned symbol is to use INTERN, e.g. (INTERN "FOO"). INTERN interns the symbol in the current package by default, but one can pass a package as second argument. After that, (FIND-SYMBOL "FOO") will return our interned symbol as expected.

More surprising, the reader automatically interns symbols. You can test it by evaluating (READ-FROM-STRING "BAR"). After evaluation, BAR is a symbol interned in the current package. This also means that it is very easy to pollute a package with symbols in ways you did not necessarily expect. To clean up, simply use UNINTERN. Remember to refer to the right symbol: to remove the symbol BAR from the package FOO, use (UNINTERN 'FOO::BAR "BAR").

A symbol is either internal or external. EXPORT will make a symbol external to its package while UNEXPORT will make it internal. As for UNINTERN, confusion usually arises around which symbol is affected. (UNEXPORT 'FOO:BAR "FOO") correctly refers to the external symbol in the FOO package and makes it internal again. (UNEXPORT 'BAR "FOO") will signal an error since the BAR symbol is not part of the FOO package (unless of course the current package happens to be FOO).

Packages themselves can be created with MAKE-PACKAGE and destroyed with DELETE-PACKAGE. Developers are usually more familiar with DEFPACKAGE, a macro allowing the creation of a package and its configuration (package use list, imported and exported symbols, etc.) in a declarative way. A surprising and frustrating behavior is that evaluating a DEFPACKAGE form for a package that already exists will result in undefined behavior if the new declaration “is not consistent” (CLtL2 11.7) with the current state of the package. As an example, adding symbols to the export list is perfectly fine. Removing one will result in undefined behavior (usually an error) due to the inconsistency of the export list. Fortunately, Common Lisp offers all the necessary functions to manipulate packages and their symbols: use them!

Classes

The Common Lisp standard includes CLOS, the Common Lisp Object System. Unsurprisingly it provides multiple ways to interact with classes and objects dynamically.

As variables or functions, classes are identified by symbols and FIND-CLASS returns the class associated with a symbol. Class names are part of a separate namespace shared with structures and types.

The DEFCLASS macro is the only way to define or redefine a class. Redefining a class means that instances created afterward with MAKE-INSTANCE will use the new definition. Existing instances are updated: newly added slots are added (either unbound or using the value associated with :INITFORM) and slots that are not defined anymore are deleted. UPDATE-INSTANCE-FOR-REDEFINED-CLASS is particularly interesting: developers can define methods for this generic function in order to control how instances are updated when their class is redefined.

Defining classes may imply implicitly defining methods: the :ACCESSOR, :READER and :WRITER slot keyword arguments will lead to the creation of generic functions. When a class is redefined, methods associated with slots that have been removed will live on.

A limitation of CLOS is that classes cannot be deleted. FIND-CLASS can be used as a place, and (SETF (FIND-CLASS 'FOO) NIL) will remove the association between the FOO symbol and the class, but the class itself and its instances will not disappear. While this limitation may seem strange, ask yourself how an implementation should handle instances of a class that has been deleted.

The class of an instance can be changed with CHANGE-CLASS: slots that exist in the new class will be conserved while those that do not are deleted. New slots are either unbound or set to the value associated with :INITFORM in the new class. In a way similar to UPDATE-INSTANCE-FOR-REDEFINED-CLASS, UPDATE-INSTANCE-FOR-DIFFERENT-CLASS lets developers control precisely the process.

Generics and methods

Generics are functions which can be specialized based on the class (and not type as one could expect) of their arguments and which can have a method combination type.

Generics can be created explicitly with DEFGENERIC or implicitly when DEFMETHOD is called and the list of parameter specializers and method combination does not match any existing generic function. Since generics are functions, FBOUNDP, SYMBOL-FUNCTION and FMAKUNBOUND will work as expected.

Methods themselves are either defined as part of the DEFGENERIC call or separately with DEFMETHOD. Discovering the different methods associated with a generic function is a bit more complicated. There is no standard way to list the methods associated with a generic, but it is at least possible to look up a method with FIND-METHOD. Do remember to pass a function (and not a symbol) as the generic, and to pass classes (and not symbols naming classes) in the list of specializers.

Redefinition is not as obvious as for non-generic functions. When redefining a generic with DEFGENERIC all methods defined as part of the previous DEFGENERIC form are removed and methods defined in the redefinition are added. However, methods defined separately with DEFMETHOD are not affected.

For example, in the following code, the second call to DEFGENERIC will replace the two methods specialized on INTEGER and FLOAT respectively by a single one specialized on a STREAM, but the method specialized on STRING will remain unaffected.

(defgeneric foo (a)
  (:method ((a integer))
    (format nil "~A is an integer" a))
  (:method ((a float))
    (format nil "~A is a float" a)))

(defmethod foo ((a string))
  (format nil "~S is a string" a))

(defgeneric foo (a)
  (:method ((a stream))
    (format nil "~A is a stream" a)))

Note that trying to redefine a generic with a different parameter lambda list will cause the removal of all previously defined methods since none of them can match the new parameters.

Removing a method will require you to find it first using FIND-METHOD and then use REMOVE-METHOD. With the previous example, removing the method specialized on a STRING argument is done with:

(remove-method #'foo (find-method #'foo nil (list (find-class 'string)) nil))

Working with methods is not always easy, and two errors are very common.

First, remember that changing the combinator in a DEFMETHOD will define a new method. If you realize that your :AFTER method should use :AROUND and reevaluate the DEFMETHOD form, remember to delete the method with the :AFTER combinator or you will end up with two methods being called.

Second, when defining a method for a generic from another package, remember to correctly refer to the generic. If you want to define a method on the BAR generic from package FOO, use (DEFMETHOD FOO:BAR (…) …) and not (DEFMETHOD BAR (…) …). In the latter case, you will define a new BAR generic in the current package.

Meta Object Protocol

While CLOS is already quite powerful, various interactions are impossible. One cannot create classes or methods programmatically, introspect classes or instances for example to list their slots or obtain all their superclasses, or list all the methods associated with a generic function.

In addition of an example of a CLOS implementation, The Art of the Metaobject Protocol² defines multiple extensions to CLOS including metaclasses, metaobjects, dynamic class and generic creation, class introspection and much more.

Most Common Lisp implementations implement at least part of these extensions, usually abbreviated as “MOP”, for “MetaObject Protocol”. The well-known closer-mop system can be used as a compatibility layer for multiple implementations.

Structures

Structures are record constructs defined with DEFSTRUCT. At a glance they may seem very similar to classes, but they have a fundamental limitation: the results of redefining a structure are undefined (CLtL2 19.2).

While this property allows implementations to handle structures in a more efficient way than classes, it makes structures unsuitable for incremental development. As such, they should only be used as a last resort, when a regular class has been proved to be a performance bottleneck.

Conditions

While conditions look very similar to classes the Common Lisp standard does not define them as classes. This is one of the few differences between the standard and CLtL2 which clearly states in 29.3.4 that “Common Lisp condition types are in fact CLOS classes, and condition objects are ordinary CLOS objects”.

This is why one uses DEFINE-CONDITION instead of DEFCLASS and MAKE-CONDITION instead of MAKE-INSTANCE. This also means that one should not use slot-related functions (including the very useful WITH-SLOTS macro) with conditions.

In practice, most modern implementations follow CLtL2 and the CLOS-CONDITIONS:INTEGRATE X3J13 Cleanup Issue and implement conditions as CLOS classes, meaning that conditions can be manipulated and redefined as any other classes. And the same way as any other classes, they cannot be deleted.

Types

Types are identified by symbols and are part of the same namespace as classes (which should not be surprising since defining a class automatically defines a type with the same name).

Types are defined with DEFTYPE, but documentation is surprisingly silent on the effects of type redefinition. This can lead to interesting situations. On some implementations (e.g. SBCL and CCL), if a class slot is defined as having the type FOO, redefining FOO will not be taken into account and the type checking operation (which is not mandated by the standard) will use the previous definition of the type. Infortunatly Common Lisp does not mandate any specific behavior on slot type mismatches (CLtL2 28.1.3.2).

Thus developers should not expect any useful effect from redefining types. Restarting the implementation after substantial type changes is probably best.

In the same vein interactions with types are very limited. You cannot find a type by its symbol or even check whether a type exists or not. Calling TYPE-OF on a value will return a type this value satisfies, but the nature of the type is implementation-dependent (CLtL2 4.9): it could be any supertype. In other words, TYPE-OF could absolutly return T for all values but NIL. At least SUBTYPE-P lets you check whether a type is a subtype of another type.

Going further

Common Lisp is a complex language with a lot of subtleties, way more than what can be covered in a blog post. The curious reader will probably skip the standard (not because you have to buy it, but because it is a low quality scan of a printed document and jump directly to CLtL2 or the Common Lisp HyperSpec. The Art of the Metaobject Protocol is of course the normative reference for the CLOS extensions usually referred to as “MOP”.

Guy L. Steele Jr. Common Lisp the Language, 2nd edition. 1990. ↩︎
Gregor Kiczales, Jim des Rivieres and Daniel G. Bobrow. The Art of the Metaobject Protocol. 1991. ↩︎

Building Erlang applications the hard way

nicolas@n16f.net (Nicolas Martyanoff) — Sat, 17 Jun 2023 18:00:00 +0000

Edit (2023-06-19)
Updated to mention the +Bd option.

I was recently in a conversation involving Erlang, and it made me feel like revisiting the subject. Last time I used Erlang for a commercial application, I remember a lot of frustration with the tooling, due in part to the small size of the ecosystem. Today I decided to read the documentation and see how to build an Erlang program from scratch. Forget about Rebar3 and Erlang.mk, we are going to do this the hard way.

A minimal Erlang program

Let us start with a small program. We could of course just write a traditional Hello World printing a message and exiting, but we are using Erlang and Erlang is all about servers. So our program is a server printing a message to the standard output every second. We will build it according to OTP principles.

Writing the modules

We will need three modules.

First the application, whose only role is to start the supervisor:

-module(hello_app).

-behaviour(application).

-export([start/2, stop/1]).

start(_StartType, _Args) ->
  hello_sup:start_link().

stop(_State) ->
  ok.

Then the supervisor, with a single child:

-module(hello_sup).

-behaviour(supervisor).

-export([start_link/0]).
-export([init/1]).

start_link() ->
  supervisor:start_link(?MODULE, []).

init(_Args) ->
  Children = [#{id => hello_server,
                start => {hello_server, start_link, []}}],
  {ok, {#{}, Children}}.

And finally the server, printing a message every second:

-module(hello_server).

-behaviour(gen_server).

-export([start_link/0]).
-export([init/1, terminate/2, handle_call/3, handle_cast/2, handle_info/2]).

start_link() ->
  gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).

init([]) ->
  schedule_hello(),
  {ok, undefined}.

terminate(_Reason, _State) ->
  ok.

handle_call(_Msg, _From, State) ->
  {reply, unhandled, State}.

handle_cast(_Msg, State) ->
  {noreply, State}.

handle_info({hello, Message}, State) ->
  io:format("~ts~n", [Message]),
  schedule_hello(),
  {noreply, State}.

schedule_hello() ->
  erlang:send_after(1000, self(), {hello, <<"Hello world!">>}).

We store module files in a src directory.

Writing the application definition

In OTP, an application is a component made of a set of modules working together. Applications are defined with an application file; they contain an Erlang expression describing the application and end with the .app extension. OTP, more precisely the application controller, will use it to know how to start, stop and in general handle the application.

Since the application file is read at runtime, we will store it in a second directory named ebin, which will also be used to store BEAM code files.

Add the following content to ebin/hello.app:

{application, hello,
 [{description, "A hello world application."},
  {vsn, "1.0.0"},
  {modules,
   [hello_app,
    hello_sup,
    hello_server]},
  {registered,
   [hello_sup,
    hello_server]},
  {applications,
   [kernel,
    stdlib]},
  {mod, {hello_app, []}},
  {env, []}]}.

Again nothing complicated. Feel free to refer to the documentation for more information.

Compiling and running the application

At the lowest level, we need to be able to compile modules, transforming Erlang code into BEAM bytecode. We can do that just by calling erlc. Of course we probably do not want to call it manually for each module after each modification, so let us write a simple Makefile:

ERLC_OPTIONS ?= -Wall -Werror

ERL_SRC = $(wildcard src/*.erl)
ERL_OBJ = $(patsubst src/%.erl,ebin/%.beam,$(ERL_SRC))

all: build

build: $(ERL_OBJ)

ebin/%.beam: src/%.erl
	erlc $(ERLC_OPTIONS) -o $(dir $@) $<

clean:
	$(RM) $(wildcard ebin/*.beam)

.PHONY: all build clean

Careful about the -o option: while most compilers use it to provide the name of the destination file, erlc expects a directory but will not signal an error if it is a file, instead ignoring the option altogether.

Run make to compile your modules. We can now run our application. Start erl, using the -pa ebin option to add the ebin directory to the list of paths Erlang looks for code files, and start the hello application in the shell:

application:start(hello).

Barring any unexpected error, you should see the Hello world! message printed every second.

Of course you can also run it in a non-interactive way:

erl -noshell -pa ebin -eval "application:start(hello)"

Note that even with -noshell, there does not seem to be any way to tell the Erlang runtime to exit on SIGINT instead of opening a interactive “break handler” requiring additional input. Disappointing. You will notice that using C-c in a terminal, i.e. sending SIGINT to the Erlang runtime, starts the interactive break handler instead of stopping the program. You can use the -Bd option to exit on SIGINT like other programs. Thanks to Asabil for telling me about the -B option on Reddit!

At this point it is tempting to call it a day. After all we can define an application, build its modules and run it. But this approach is limited: we still need to install Erlang on the target server, ideally with the same version as the one used for development and tests. We could also be tempted to use code hot-swapping to upgrade the application without stopping it.

Erlang supports releases, a way of packaging multiple applications and the Erlang runtime itself into a self-contained archive. So let us build a release!

Building a release

Creating the release resource file

A release starts with a release resource file, an Erlang file describing the release and the application it will contain.

Our release is simple:

{release, {"hello", "1.0.0"},
 {erts, "13.2"},
 [{kernel, "8.5.4"},
  {stdlib, "4.3"},
  {hello, "1.0.0"}]}.

We define a released named hello with version 1.0.0, based on ERTS 13.2, and we list three applications: kernel, stdlib, and hello. Remember that kernel and stdlib must always be included.

You will notice that we need to provide the version for each component. This is really frustrating because they depend on the version of Erlang installed on your system, and you would expect systool to handle it automatically, but this is how it is. Clearly the release resource file is something to be generated automatically. In the mean time, you will need to adapt the version numbers of the example (those are for Erlang 25.3.2).

Generating the boot script

While the release resource file defines what is in the release, the boot script explains how to start it. Again, it contains Erlang expressions, and it can quickly get complicated (see the documentation for more information). Furthermore, the Erlang release handler will not use the boot script directly, but instead a compiled version with the .boot file extension. Fortunately we do not have to write the script ourselves: systools will handle both generation and compilation.

To use systools, let us write a small Erlang program named generate-boot:

#!/usr/bin/env escript

main([ReleaseName]) ->
  Options = [no_warn_sasl, {path, ["ebin"]}, silent],
  case systools:make_script(ReleaseName, Options) of
    {ok, _Module, []} ->
      ok;
    {ok, Module, Warnings} ->
      log_error("warning: ~ts", [Module:format_warning(Warnings)]);
    {error, Module, Errors} ->
      log_error("error: ~ts", [Module:format_error(Errors)]),
      halt(1)
  end;
main(_Args) ->
  log_error("Usage: ~ts <release-name>~n", [escript:script_name()]),
  halt(1).

log_error(Format, Args) ->
  io:format(standard_error, Format, Args).

We expect the script to be called with one argument being the name of the release, and all we do is to call systools:make_script to generate the script file and compile it to a boot file.

We provide three options:

no_warn_sasl to tell systools not to print any warning because we did not include the SASL application (unnecessary since we do not deal with release upgrading in this article).
{path, ["ebin"]} to look in the ebin directory for application definition files.
silent to return warnings and errors instead of printing them directly. This is necessary because we want to log to the standard error output and exit with status code 1.

We then update the Makefile to be able to build the boot file:

RELEASE = hello

$(RELEASE).boot: $(RELEASE).rel ebin/$(RELEASE).app
	./generate-boot $(RELEASE)

Generating a release package

The final form of a release is a tarball containing all the files required to run our program, and again systools knows how to generate it. So let us write another Erlang program named generate-tarball:

#!/usr/bin/env escript

main([ReleaseName]) ->
  ERTSPath = code:root_dir(),
  Options = [no_warn_sasl, {path, ["ebin"]}, {erts, ERTSPath}, silent],
  case systools:make_tar(ReleaseName, Options) of
    {ok, _Module, []} ->
      ok;
    {ok, Module, Warnings} ->
      log_error("warning: ~ts", [Module:format_warning(Warnings)]);
    {error, Module, Errors} ->
      log_error("error: ~ts", [Module:format_error(Errors)]),
      halt(1)
  end;
main(_Args) ->
  log_error("usage: ~ts <release-name>~n", [escript:script_name()]),
  halt(1).

log_error(Format, Args) ->
  io:format(standard_error, Format, Args).

The code is very similar to the one we wrote to generate the boot script. We introduce a new option, {erts, ERTSPath}, to tell systools to add the Erlang runtime to the release. This way, our release will be fully standalone: we will be able to run it anywhere without having to install Erlang. This option requires the path containing Erlang libraries, and we obtain it with the code:root_dir function.

Then we add the new release target to the Makefile:

release: $(RELEASE).tar.gz

$(RELEASE).tar.gz: $(ERL_OBJ) $(RELEASE).boot
	./generate-tarball $(RELEASE)

At this point, we can simply run make release, and let GNU Make create the archive.

Running the application

Now that we have a tarball, we can extract it somewhere and start Erlang:

mkdir /tmp/hello
tar -xf hello.tar.gz -C /tmp/hello
cd /tmp/hello
./erts-13.2/bin/erl -noshell -boot releases/1.0.0/start

As you can see, the release contains Erlang itself. We start it with our boot script, which has been stored in releases/1.0.0/start.boot, and our application starts.

Conclusion

We can now build an Erlang release from scratch! Obviously a real world application would have other requirements such as handling multiple applications (internal or added from external dependencies), tests, several build profiles, extra data files or scripts to be added the release… And while it can be handled with systools and a few scripts, you probably want to start looking at either Rebar3 or Erlang.mk.

But you should now have a better understanding of the whole build and release process. As usual, the best way to figure out how something works is to get rid of the high level tools and try to do it on your own.

Reduce vs fold in Common Lisp

nicolas@n16f.net (Nicolas Martyanoff) — Fri, 02 Jun 2023 18:00:00 +0000

Introduction

If you have already used functional languages, you are probably familiar with fold, a high order function used to iterate on a collection of values to combine them and return a result. You may be surprised that Common Lisp does not have a fold function, but provides REDUCE which works a bit differently. Let us see how they differ.

Understanding `REDUCE`

In its simplest form, REDUCE accepts a function and a sequence (meaning either a list or a vector). It then applies the function to successive pairs of sequence elements.

You can easily check what happens by tracing the function:

CL-USER> (trace +)
CL-USER> (reduce #'+ '(1 2 3 4 5))
  0: (+ 1 2)
  0: + returned 3
  0: (+ 3 3)
  0: + returned 6
  0: (+ 6 4)
  0: + returned 10
  0: (+ 10 5)
  0: + returned 15
15

In this example, the call to REDUCE evaluates (+ (+ (+ (+ 1 2) 3) 4) 5).

You can reverse the order using the :from-end keyword argument:

CL-USER> (trace +)
CL-USER> (reduce #'+ '(1 2 3 4 5) :from-end t)
  0: (+ 4 5)
  0: + returned 9
  0: (+ 3 9)
  0: + returned 12
  0: (+ 2 12)
  0: + returned 14
  0: (+ 1 14)
  0: + returned 15
15

In which case you will evaluate (+ 1 (+ 2 (+ 3 (+ 4 5)))). The result is of course the same since the + function is associative.

You can of course provide an initial value, in which case REDUCE will behave as if this value has been present at the beginning (or the end with :from-end) of the sequence.

The surprising aspect of REDUCE is its behaviour when called on a sequence with less than two elements:

If the sequence contains a single element:
- if there is no initial value, the function is not called and the element is returned directly;
- if there is one, the function is called on both the initial value and the single element.
If the sequence is empty:
- if there is no initial value, the function is called without any argument;
- if there is one, the function is not called and the initial value is returned directly.

As a result, any function passed to REDUCE must be able to handle being called with zero, one or two arguments. Most examples found on the Internet use + or append, and these functions happen to handle it (e.g. (+) returns the identity element of the addition, zero). If you write your own functions, you will have to deal with it using the &OPTIONAL lambda list keyword.

This can lead to unexpected behaviours. If you compute the sum of a sequence of floats using (reduce #'+ floats), you may find it logical to obtain a float. But if FLOATS is an empty sequence, you will get 0 which is not a float. Something to keep in mind.

Differences with fold

The fold function is traditionally defined as accepting three arguments: a function, an initial value — or accumulator — and a list. The function is called repeatedly with both the accumulator and a list element, using the value returned by the function as next accumulator.

For example in Erlang:

lists:foldl(fun(X, Sum) -> Sum + X end, 0, [1, 2, 3, 4, 5]).

An interesting consequence is that fold functions are always called with the same type of arguments (the list value and the accumulator), while REDUCE functions can be called with zero or two list values. This makes it harder to write functions when the accumulated value has a different type from sequence values.

Fold is also simpler than REDUCE since it does not have any special case, making it easier to reason about its behaviour.

It would be interesting to know why a function as fundamental as fold was not included in the Common Lisp standard.

Implementing `FOLDL`

We can of course implement a fold function in Common Lisp. We will concentrate on the most common (and most efficient) left-to-right version. Let us start by a simple implementation for lists:

(defun foldl/list (function value list)
  (declare (type (or function symbol) function)
           (type list list))
  (if list
      (foldl/list function (funcall function value (car list)) (cdr list))
      value))

As clearly visible, the recursive call to FOLDL/LIST is in tail position and SBCL will happily perform tail-call elimination.

For vectors we use an iterative approach:

(defun foldl/vector (function value vector)
  (declare (type (or function symbol) function)
           (type vector vector))
  (do ((i 0 (1+ i))
       (accumulator value))
      ((>= i (length vector))
       accumulator)
    (setf accumulator (funcall function accumulator (aref vector i)))))

Finally we write the main FOLDL function which operates on any sequence:

(defun foldl (function value sequence)
  (declare (type (or function symbol) function)
           (type sequence sequence))
  (etypecase sequence
    (list (foldl/list function value sequence))
    (vector (foldl/vector function value sequence))))

At the point we can already use FOLDL for various operations. We could of course improve it with the addition of the usual :START, :END and :KEY keyword arguments for more flexibility.

Replacing Ngrok with Frp

nicolas@n16f.net (Nicolas Martyanoff) — Thu, 20 Apr 2023 18:00:00 +0000

I have been using Ngrok for almost two years in order to test webhooks during development. It works really well, true; but even the personal plan costs $96 a year if you want a fixed URI. I recently came across Frp, a reverse proxy packed with useful features. This is how I used it to replace Ngrok entirely.

Of course you will need a server running somewhere. It does not have to be powerful, even the smallest VPS will do the work just fine. Personally I use FreeBSD, but Frp is written in Go and should run anywhere.

Configuring the server

Frp server

Server configuration turned out to be more complicated than expected. In the current state, the FreeBSD package runs Frp as the nobody user, meaning that it cannot read external files owned by root such as a TLS key. In a perfect world, Frp would start as root, would read the TLS key and certificate then would switch to a non-privileged user. Infortunately Frp does not support that.

Of course we could let Frp run as root, but there is a much better solution. Since I already use NGINX, what about simply letting it handle TLS? This way Frp can run as a non-privileged user without any issue.

Let’s make it happen! First we configure the Frp server; the configuration (located at /usr/local/etc/frps.ini on FreeBSD) is minimal:

[common]
bind_addr = 127.0.0.1
bind_port = 6001

proxy_bind_addr = 127.0.0.1
vhost_http_port = 8042

authentication_method = token
token = 12345678

We bind both the main interface and the proxy interface to a local address since we are going to use NGINX as front. We also require token authentication from any Frp client. Of course do not forget to replace 12345678 by a proper random password.

You may also want to patch the rc script at /usr/local/etc/rc.d/frps to run Frps as a dedicated user, and make sure that the configuration file is owned by it with permissions 0600.

NGINX

For NGINX, we need a more complex configuration.

We first need a block to define a HTTPS interface which will be used to access what is running on our workstation:

https {
    server {
        listen 443 ssl;
        listen [::]:443 ssl;

        server_name dev.example.com;

        ssl_certificate /etc/ssl/certs/dev.example.com.crt;
        ssl_certificate_key /etc/ssl/private/dev.example.com.key;

        access_log /var/log/nginx/access-dev.log combined;

        location / {
            proxy_pass http://localhost:8042;

            proxy_set_header Host $host;
            proxy_set_header X-Request-Id $request_id;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }
}

Obviously you will have to adjust the server name and SSL certificate file paths (my certificate comes from Let’s Encrypt and is generated by Lego).

Then we need a TCP interface to connect to the control port of the Frps server. NGINX can do just that with the stream module. On FreeBSD it is compiled as a dynamic library, so we have to load it first.

load_module /usr/local/libexec/nginx/ngx_stream_module.so;

stream {
    upstream frps {
        server 127.0.0.1:6001;
    }

    server {
        listen 6000 ssl;

        ssl_certificate /etc/ssl/certs/dev.example.com.crt;
        ssl_certificate_key /etc/ssl/private/dev.example.com.key;

        proxy_pass frps;
    }
}

I was positively surprised by how simple it is. Again, NGINX saves the day.

The resulting architecture is a bit convoluted, but it works.

Configuring the client

As Ngrok, Frp requires a client to run on your workstation. Fortunately it is already available on AUR for Archlinux, so installing it was trivial:

git clone https://aur.archlinux.org/frp.git
cd frp
makepkg
sudo pacman -U ./frpc-0.48.0-1-x86_64.pkg.tar.zst

The default path for the configuration file is ./frpc.ini which is a really bad idea. Ideally you would want a standard absolute location such as ~/.frpc.ini or ~/.config/frpc.ini. But we work we what we have.

I went with a small script to run the client, and stored it at ~/bin/frpc since ~/bin/ is my PATH environment variable.

#!/bin/sh

exec /usr/bin/frpc --config $HOME/.frpc/frpc.ini "$@"

This way I can run frpc without having to worry about the path to the configuration file. And in the future, if I end up needing several configurations, I can easily add options to the script to select the right file.

The configuration itself is quite simple:

[common]
server_addr = server.example.com
server_port = 6000

tls_enable = true
disable_custom_tls_first_byte = true

authentication_method = token
token = 12345678

[web]
type = http
local_port = 8080
custom_domains = dev.example.com

In the common section, we indicate how to contact the Frp server and how to authenticate. You will notice the strange disable_custom_tls_first_byte option. Frp uses a single byte sent by the client to identify whether the connection is going to use TLS or not, allowing it to support both TLS and non-TLS connections on the same port. Since we use NGINX as TLS interface, we need to instruct the client not to send this first byte.

In the web section, we declare our web proxy: we will access our local application on dev.example.com, and we expect it to listen locally on port 8080. Obviously you will have to choose a domain you control and make sure it points to the Frp server.

Done!

At this point, the only thing left is to start the Frp client on your workstation. It will connect to the Frp server, and you will then be able to publicly access your local application from the domain specified.

Communication between the Frp client and server is secured with TLS, and the application is exposed using HTTPS, both thanks to NGINX. All good!

You should also have a look at the Frp readme because it can do much more than just proxying HTTP traffic. OIDC authentication, builtin admin interface, Prometheus metrics, load balancing, you probably will find something useful to you.

Making IELM More Comfortable

nicolas@n16f.net (Nicolas Martyanoff) — Sat, 08 Apr 2023 18:00:00 +0000

IELM is the Emacs Lisp REPL. It lets you evaluate any Elisp expression, prints the output and keeps track of previous commands. While it is a serious step-up from the humble eval-expression, it can be uncomfortable on several aspects. Let us improve it.

Adding Eldoc hints

Eldoc is a generic module used to display realtime documentation hints while you type. For Emacs Lisp, it means displaying docstrings when your cursor is over a symbol or when you start writing a function call.

When calling a function, you often need a quick remainder about the arguments. Having Eldoc means you do not have to call C-h f and deal with the documentation buffer, and can just glance at the echo area.

To enable Eldoc in IELM, let us add it to the initialization hook:

(add-hook 'ielm-mode-hook 'eldoc-mode)

Using Paredit

Paredit is an essential minor mode that lets you manipulate Lisp code as expressions and not as simple text. If you are not using it and think that editing all these parentheses is annoying, you are missing out. Give it a try, you will not be disappointed.

To use Paredit in IELM, we add it to the initialization hook:

(add-hook 'ielm-mode-hook 'paredit-mode)

Infortunately there is a key conflict between Paredit and IELM. Paredit overrides return to execute paredit-RET, meaning that the original ielm-return is not called.

The simplest way to fix it is to alter the Paredit keymap:

(define-key paredit-mode-map (kbd "RET") nil)
(define-key paredit-mode-map (kbd "C-j") 'paredit-newline)

We remove the entry associated with return and use C-j to insert a newline character, useful to write a multiline expression.

Making the command history persistent

The most annoying part of IELM is that it does not have persistent history. I expect any interactive command system to store past entries since they can always be useful again. ZSH has persistence, so does the SLIME Common Lisp REPL. IELM does not, even though Comint (the mode IELM derives from) can handle it.

Let us add it. First we will write a function to configure Comint and load any entry stored in the history file if it exists:

(defun g-ielm-init-history ()
  (let ((path (expand-file-name "ielm/history" user-emacs-directory)))
    (make-directory (file-name-directory path) t)
    (setq-local comint-input-ring-file-name path))
  (setq-local comint-input-ring-size 10000)
  (setq-local comint-input-ignoredups t)
  (comint-read-input-ring))

We store the history in the ielm/history file in the user Emacs directory, mimicking the eshell/history file used by Eshell.

Nothing complicated: we increase the number of entries stored in the history file, and tell Comint to drop duplicate entries. We also are careful and use setq-local to make sure Comint settings are only modified for the current buffer and not globally, since other buffers using different Comint-based modes could have different requirements.

We want to call this function when IELM start:

(add-hook 'ielm-mode-hook 'g-ielm-init-history)

Reading the history is one thing. We need a way to add all expressions we evaluate to it. It would have been nice to have a hook called everytime an expression is entered, but we have to do without it. We are going to use Elisp advices to evaluate code each time a specific function is called. We want to write the history file when ielm-send-input returns:

(defun g-ielm-write-history (&rest _args)
  (with-file-modes #o600
    (comint-write-input-ring)))
  
(advice-add 'ielm-send-input :after 'g-ielm-write-history)

Advising functions are called with the same arguments as the advised function. We do not care about them, so we use the &rest keyword to match all arguments passed to g-ielm-write-history.

We are careful to use with-file-modes to make sure the file is always created as only readable by the user, since it may contain private information.

This was not easy, but persistent history is worth it!

Useful key bindings

Finally we will bind two key combinations.

First the very common C-l to clear the buffer. All modes deriving from Comint have C-c M-o, but it is awkward to type and C-l is very common in various applications.

(define-key inferior-emacs-lisp-mode-map (kbd "C-l")
            'comint-clear-buffer)

Then we want a way to search through old expressions. Comint has comint-history-isearch-backward-regexp which is incredibly primitive. We can get a far better experience with Helm. The helm-comint-input-ring function lets us browse among old expressions and select one through incremental search. We bind it to C-r:

(define-key inferior-emacs-lisp-mode-map (kbd "C-r")
            'helm-comint-input-ring)

IELM is so much more comfortable to use now!

Golang workspaces, at last!

nicolas@n16f.net (Nicolas Martyanoff) — Tue, 28 Mar 2023 18:00:00 +0000

Workspaces are a feature introduced in March 2022 with Go 1.18. They did not really get a lot of publicity, and I have not had the chance to experiment with them until recently. However I am really glad I did because they improve a major aspect of my workflow: dealing with multiple modules.

Go modules are not ideal

Go modules were a big help when they were introduced in 2018, but they were always limited. Larger products are often split into multiple projects: one or more applications and several libraries and tools, meaning multiple modules.

While module versioning makes sure that components use the right version of each dependency, it is also really annoying during development. If your application foo depends on the go-bar library, you will often have to work on go-bar while testing the changes in foo. This means updating go-bar, commiting changes, pushing them, and updating the foo module with go get. Quite cumbersome.

A first improvement is the module replacement system. In the example above, we could instruct Go to use the local copy of go-bar when building foo. Assuming that foo and go-bar are at the same level in the filesystem:

go mod edit -replace example.com/myproject/go-bar=../go-bar

With this simple change, you have made your life easier: you can work on changes in go-bar, and immediately build and run foo without any additional operations. When you are done, you can still commit and push normally in go-bar.

But it is still not perfect. Module replacements are stored in the go.mod file, meaning that these changes will be pushed to your central repository circumventing dependency versioning and causing issues for other developers and CI processes. While module replacement works fine to point a module to a fork, it is not really a solution for our problem.

Using workspaces

Go workspaces let you create environments where you control the source of the modules you use, without having to modify these modules.

Going back to our example, we can solve our problem by creating a workspace for foo in which go-bar refers to the local copy. In the directory of foo:

go work init
go work use .
go work use ../go-bar

Nothing complicated here. The init subcommand creates the go.work file which will contain the configuration of the workspace. Then the use subcommand is called to add two modules to the workspace: the foo module in the current repository, i.e. ., and the one in the go-bar sibling directory.

At this point, building foo will correctly use the local copy of go-bar without having to modify any of the modules. Problem solved.

Even better, you can include replace declarations in the go.work file the exact same way as in go.mod file. These declarations will override those in module files, giving you total control on the environment, again without having to alter modules.

Committing the workspace file

Whether you commit the workspace file or not depends on your situation. When working in a mono-repository with other developers, committing a workspace file allows everyone to build the project the exact same way, using dependencies in the repository. It can also be practical with multiple repositories as long as you expect everyone to organize their local copies the same way.

But you can also keep your own workspace files without committing them. This gives you the ability to quickly switch to a local copy for a dependency or to replace a module by another one during development. For example this what I do for my go-raft library project. The program in the cmd/kvstore directory is based on the go-service library. During development, I sometimes have to add code to go-service. Therefore I have a workspace file in go-raft which references the local copy of go-service. But I do not commit it, to avoid affecting anyone trying to build go-raft.

This flexibility is really practical, and I’m quite satisfied with workspaces at this point.

What could be better

There is a small issue to keep in mind. If you get used to work with workspaces, committing your local dependencies and having your program use them automatically, you may still have to maintain module dependencies. It is not necessarily a problem if you commit the workspace file and always use it, but it makes sense to keep dependencies up-to-date in your module files.

The go work command has a sync subcommand whose description let me think that it would update modules included in the workspace to the right dependency versions, but it turns out not do to so when tracking pseudo-versions (i.e. when you are tracking a branch and not a specific tag).

Therefore I have to continue to update non-tagged dependencies with this usual (and quite excessive) command, here for go-service:

GOPROXY=direct go get github.com/galdor/go-service@latest
go mod tidy

This way I make sure to bypass the Go proxy, fetch the last version of the master branch and clean up the dependency list.

Despite this small inconvenience, Go workspaces have made my daily work much easier. Still a win!

Counting lines with Common Lisp

nicolas@n16f.net (Nicolas Martyanoff) — Fri, 17 Mar 2023 18:00:00 +0000

A good line counting program has two features: it only counts non-empty lines to get a fair estimate of the size of a project, and it groups line counts by file type to help see immediately which languages are used.

A long time ago I got frustrated with two well known line counters. Sloccount spits out multiple strange Perl warnings about locales, and most of the output is a copyright notice and some absurd cost estimations. Cloc has fourteen Perl packages as dependencies. Writing a simple line counter is an interesting exercise; at the time I was discovering Common Lisp, so I wrote my own version.

I made a few changes years after years, but most of the code stayed the same. I thought it would be interesting to revisit this program and present it part by part as a demonstration of how you can use Common Lisp to solve a simple problem.

We are going to write the program bottom-up, starting with the smallest building blocks and progressively building upon them.

The program

The program is written in Common Lisp. The most convenient way of storing and executing it is a single executable file stored in a directory being part of the PATH environment variable. In my case, the script will be called locc, for “line of code counter”, and will be stored in the ~/bin directory.

We start the file with a shebang indicating how to execute the file. We use the SBCL implementation because it is stable and actively developed. It also makes it easy to execute a simple file:

#!/usr/bin/sbcl --script

Finding files

Our line counter will operate on directories, so it has to be able to list files in them. Path handling functions are very disconcerting at first. Common Lisp was designed a long time ago, and operating systems were different at the time. Let us dig in!

First let us write a simple function to check if a pathname object is a directory pathname:

(defun directory-path-p (path)
  "Return T if PATH is a directory or NIL else."
  (declare (type (or pathname string) path))
  (and (not (pathname-name path))
       (not (pathname-type path))))

Then we write a function to identify hidden files and directories since we do not want to include them:

(defun hidden-path-p (path)
  "Return T if PATH is a hidden file or directory or NIL else."
  (declare (type pathname path))
  (let ((name (if (directory-path-p path)
                  (car (last (pathname-directory path)))
                  (file-namestring path))))
    (and (plusp (length name))
         (eq (char name 0) #\.))))

As you can see we use DIRECTORY-PATH-P to extract the basename of the path, then check if it starts with a full stop (only if it is not empty of course).

Finally we can write the function to actually list files in a directory recursively:

(defun directory-path (path)
  "If PATH is a directory pathname, return it as it is. If it is a file
pathname or a string, transform it into a directory pathname."
  (declare (type (or pathname string) path))
  (if (directory-path-p path)
      path
      (make-pathname :directory (append (or (pathname-directory path)
                                            (list :relative))
                                        (list (file-namestring path)))
                     :name nil :type nil :defaults path)))

(defun find-files (path)
  "Return a list of all files contained in the directory at PATH or any of its
subdirectories."
  (declare (type (or pathname string) path))
  (flet ((list-directory (path)
           (directory
            (make-pathname :defaults (directory-path path)
                           :type :wild :name :wild))))
    (let ((paths nil)
          (children (list-directory (directory-path path))))
      (dolist (child children paths)
        (unless (hidden-path-p child)
          (if (directory-path-p child)
              (setf paths (append paths (find-files child)))
              (push child paths)))))))

We use the DIRECTORY standard function with a path containing a wildcard component to list the files in a directory, and do so recursively.

Counting lines

Now that we have files, we can start counting lines. Let us first write a function to count the number of non-empty lines in a file.

(defun count-file-lines (path)
  "Count the number of non-empty lines in the file at PATH. A line is empty if
it only contains space or tabulation characters."
  (declare (type pathname path))
  (with-open-file (stream path :element-type '(unsigned-byte 8))
    (do ((nb-lines 0)
         (blank-line t))
        (nil)
      (let ((octet (read-byte stream nil)))
        (cond
          ((or (null octet) (eq octet #.(char-code #\Newline)))
           (unless blank-line
             (incf nb-lines))
           (when (null octet)
             (return-from count-file-lines nb-lines))
           (setf blank-line t))
          ((and (/= octet #.(char-code #\Space))
                (/= octet #.(char-code #\Tab)))
           (setf blank-line nil)))))))

We open the file to obtain a steam of octets, and read it octet by octet, keeping track of whether the current line is blank or not. Note how we make sure to count the last line even if it does not end with a newline character.

Reading a file one octet could be a disaster for performances. Fortunately SBCL file streams are buffered, something we can easily check by running our program with strace -e trace=openat,read. We would not rely on this property if we wanted our program to work on multiple Common Lisp implementations, but this is a non issue here.

Identifying the file type

Counting lines is one thing, but we need to identify their content. The simplest way is to do so based on the file extension.

Obviously we will want to ignore various files which are known not to contain text content, so we start by building a hash table containing these extensions:

(defparameter *ignored-extensions*
  (let ((extensions '("a" "bin" "bmp" "cab" "db" "elc" "exe" "gif" "gz"
                      "jar" "jpeg" "jpg" "o" "pcap" "pdf" "png" "ps" "rar"
                      "svg" "tar" "tgz" "tiff" "zip"))
        (table (make-hash-table :test 'equal)))
    (dolist (extension extensions table)
      (setf (gethash extension table) t)))
  "A hash table containing all file extensions to ignore.")

We then create another hash table to associate a type symbol to each known file extension:

(defparameter *extension-types*
  (let ((pairs '(("asm" . assembly) ("s" . assembly)
                 ("adoc" . asciidoc)
                 ("awk" . awk)
                 ("h" . c) ("c" . c)
                 ("hpp" . cpp) ("cpp" . cpp) ("cc" . cpp)
                 ("css" . css)
                 ("el" . elisp)
                 ("erl" . erlang)
                 ("go" . go)
                 ("html" . html) ("htm" . html)
                 ("ini" . ini)
                 ("hs" . haskell)
                 ("java" . java)
                 ("js" . javascript)
                 ("json" . json)
                 ("tex" . latex)
                 ("lex" . lex)
                 ("lisp" . lisp)
                 ("mkd" . markdown) ("md" . markdown)
                 ("rb" . ruby)
                 ("pl" . perl) ("pm" . perl)
                 ("php" . php)
                 ("py" . python)
                 ("sed" . sed)
                 ("sh" . shell) ("bash" . shell) ("csh" . shell)
                 ("zsh" . shell) ("ksh" . shell)
                 ("scm" . scheme)
                 ("sgml" . sgml)
                 ("sql" . sql)
                 ("texi" . texinfo)
                 ("texinfo" . texinfo)
                 ("vim" . vim)
                 ("xml" . xml) ("dtd" . xml) ("xsd" . xml)
                 ("yaml" . yaml) ("yml" . yaml)
                 ("y" . yacc)))
        (table (make-hash-table :test 'equal)))
    (dolist (pair pairs table)
      (setf (gethash (car pair) table) (cdr pair))))
  "A hash table containing a symbol identifying the type of a file for
  each known file extension.")

With these hash tables, the function identifying the type of a file is trivial:

(defun identify-file-type (path)
  "Return a symbol identifying the type of the file at PATH, or UNKNOWN if the
file extension is not known."
  (declare (type pathname path))
  (let ((extension (pathname-type path)))
    (unless (gethash extension *ignored-extensions*)
      (gethash extension *extension-types* 'unknown))))

Collecting file information

Up to this point, we wrote several functions without connecting them. But we now have all the building blocks we need. Let us use them to accumulate information about the files in a list of directories.

(defun collect-line-counts (directory-paths)
  "Collect the line count of all files in the directories located at one of the
paths in DIRECTORY-PATHS and return them grouped by file type as an
association list."
  (declare (type list directory-paths))
  (let ((line-counts (make-hash-table)))
    (dolist (directory-path directory-paths)
      (dolist (path (find-files directory-path))
        (handler-case
            (let ((type (identify-file-type path)))
              (when (and type (not (eq type 'unknown)))
                (let ((nb-lines (count-file-lines path)))
                  (incf (gethash type line-counts 0) nb-lines))))
          (error (condition)
            (format *error-output* "~&error while reading ~A: ~A~%"
                    path condition)))))
    (let ((line-count-list nil))
      (maphash (lambda (type nb-lines)
                 (push (cons type nb-lines) line-count-list))
               line-counts)
      line-count-list)))

We get to use all previous functions. We iterate through directory paths to find non-hidden files using FIND-FILES, then we use IDENTIFY-FILE-TYPE to obtain a type symbol and COUNT-FILE-LINES to count the number of non-empty lines in each file. Results are accumulated by file type in the LINE-COUNTS hash table. During this process, we handle errors that may occur while reading files with a message on the error output. Finally we transform the hash table into an association list and return it.

Presenting results

Executing all these functions in the Lisp REPL is quite practical during developement, but the default pretty printer is not really what you expect for the final command line tool:

So let us write a function to format this association list:

(defun format-line-counts (line-counts &key (stream *standard-output*))
  "Format the line counts in the LINE-COUNTS association list to STREAM."
  (declare (type list line-counts)
           (type stream stream))
  (dolist (entry (sort line-counts '> :key 'cdr))
    (let ((type (car entry))
          (nb-lines (cdr entry)))
      (format stream "~12A  ~8@A~%"
              (string-downcase (symbol-name type)) nb-lines))))

We print the list sorted in descending order, meaning that the file type with the most number of lines comes first. Of course the output is padded to make sure numbers are all aligned.

Finalizing the program

The only thing left to do is the entry point of the script. This is the only place where we need to call a non-standard function in order to access command line arguments. If no command line argument were passed to the script, we look for files in the current directory.

(let ((paths (or (cdr sb-ext:*posix-argv*) '("."))))
  (format-line-counts
   (collect-line-counts paths)))

Much better!

There does not seem to be any performance issue: on the Linux kernel source tree, this program is almost 11 times faster than sloccount. It would be interesting to profile the program to make sure IO is the bottleneck and improve inefficient parts, but this code is fast enough for my needs.

As you can see, it is not that hard to use Common Lisp to solve problems.

When your product idea has no market

nicolas@n16f.net (Nicolas Martyanoff) — Mon, 13 Mar 2023 18:00:00 +0000

At the end of 2021 I quitted my job to create a company with an ex-colleague. It did not work out. One year later, both the first and the second product were a failure, my business partner had quit to create a VC-funded startup, and I was left to pick up the pieces. Not fun.

Since then I have been trying to find a new product to build while looking for contract work. This is a summary of my thought process around my last idea.

It all started with Hacker News

Recently one of my articles was posted on Hacker News. It is a big deal: lots of tech people read HN, and having your content featured on the first page is really nice when you are trying to get feedback, connect with like-minded people or simply raise awareness about your work. I do not post my own content on HN (though I have been told by someone from a YC company that “everyone does it all the time”), and I spotted the submission almost by accident.

Last week I was thinking about it again; missing the submission would not have been the end of the world, but it would have been useful to receive some kind of real time notification. A simple Telegram notification would work, how hard could it be? And it was not just Hacker News. I would love to be notified if my content was being posted on other websites such as Lobste.rs. My imagination started running wild. There are a lot of things on the internet I watch from time to time. But I would really prefer a real time mechanism notification. Some services support it; most use email which is useless because I do not check my emails every 5 minutes.

I suddenly remembered one of the discussions we had during a user research interview. A consultant told us very clearly: “automation is nice and all, but sometimes all I want is an SMS”. Could it be a viable product?

PingMeWhen: the concept

The core idea is a small SaaS where users can watch for various events and be “pinged” in real time when these events occur. No code, no complex rules, just an email, SMS, Slack message or Telegram notification with a link and maybe a custom message. It would start with watching Hacker News of course: when a link to your website is being posted, or when specific keywords appear in a title or in comments. Then I would extend it to other platforms. I could then consider more complex events: security vulnerability alerts, financial data, anything happening anywhere really. My mind was buzzing with all the possibilities! I always liked the idea of letting the computer do the work and tell me when something interesting happens. This could work.

I exposed the idea to a few friends; they tought it was interesting. I found a name, PingMeWhen, and immediately bought the pingmewhen.net domain. One thing done. The technical aspect was not too hard: regularly poll various data sources on the Internet depending on what users want to watch, emit events when specific things happen, and send notifications. I was confident I could build it quickly.

Seemed solid. But it was not.

Who is going to pay for that?

Something I learned from my last two failures is that there is no point in building a product unless you have established that someone (ideally multiple someones) is going to pay for it. For real, not just “yeah probably” but with the well known “take my money!” reaction. So this weekend I started to think about the potential market. It was not great.

Your product is either B2B, where you sell to other companies, or B2C where you target simple consumers. In this case, consumers are a really tough market: there is nothing really attractive about real time notifications, and people really do not want to pay for anything. Even if I could manage to somehow market the product to them (and I have no idea how), I could not hope to have anyone pay more than a couple dollars a month. Hardly a good business model: I am alone and have no way to gather thousands of consumers.

So why would a company use this product? As some Google search showed me, companies which really benefit from reacting quickly to what happens on the Internet already have specialized tools for that. All of them have their limitations of course, but they all target specific use cases (e.g. content marketing) and are therefore much easier to sell. Specific beats generic all the time. For more technical use cases, the situation is not really different. IFTTT already lets users react to a wide range of events with way more flexibility than a simple notification. And it is so cheap I cannot start to think about how to compete with them. Furthermore IFTTT is just one of them, there are dozen of similar powerful tools, including open source ones such as Huggin.

Something is really wrong: if I have trouble figuring out who is going to buy the product, does it really make sense to build it? Probably not.

Going forward

It took me three days from the first idea to the end of this article. I usually do not mind discarding ideas, but this one frustrates me: IFTTT does not have a recipe to let me know when one of my articles is being posted, and I really do not want to manage a Huggin deployment, so I’m back to square one. I feel like if have this kind of problem, others may have too. I am just not convinced enough people would pay for it.

At least I learned a good lesson: do not buy the domain name before being sure there is something worth pursuing. 20€ for a .net domain is way more expensive that it used to be.

I imagine I will keep the idea around just in case. And I might get some interesting feedback from readers, who knows.

Gmail backups using ISync

nicolas@n16f.net (Nicolas Martyanoff) — Thu, 02 Mar 2023 18:00:00 +0000

Like so many people, I use Gmail. There are not a lot of options, and Google Workspace provides me additional useful services and lets me use my own domain name. However there has been a lot of horror stories about users losing access to their Gmail account without any recourse. It is not clear if it can happen with Google Workspace, but the risk is too high to ignore.

I already backup my passwords; now is the time to backup my emails. While Google Workspace offers an export system, I wanted something scriptable. I went with IMAP access since I already use it to read emails with Gnus.

Authentication

Gmail IMAP access requires an application password. You can refer to the Google help center for more information about how to create them.

We will store credentials in the semi-standard netrc file. For Gmail, add the following entry to ~/.netrc:

machine imap.gmail.com port 993 login <email-address> <password>

Make sure that your netrc file is only user readable. Its permissions should be set to 0600.

ISync

We will export emails using the synchronization features of ISync. Note that ISync is the name of the project while mbsync is the name of the executable.

The configuration of the program is stored in ~/.mbsyncrc and is based on two main concepts: stores and channels. Stores designate location containing emails; channels describe a link between two stores: the “far” one an the “near” one.

Let us first define the store for the Gmail account:

IMAPStore gmail
Host imap.gmail.com
Port 993
SSLType IMAPS
User <email-address>
PassCmd "perl -ne '/^machine imap\\.gmail\\.com .*?login <email-address> .*?password (\\S+)/ && print \"$1\\n\"' ~/.netrc"
PipelineDepth 1

While you can use simple use Pass <password>, we will use PassCmd to instruct mbsync to extract the password from our netrc file. I first tried to use AWK for this purpose, but it does not support regexp capture, so I went with Perl. Note that the login, i.e. the email address for Gmail, has to be properly escaped in the regular expression. For example, bob@example.com must be included as bob\\@example\\.com.

PipelineDepth controls the number of IMAP commands send in a row before reading responses. It is recommended to use 1 due to Gmail rate limiting.

We then define the local store, i.e. a Maildir directory on our machine:

MaildirStore local
Path ~/mail/
Inbox ~/mail/INBOX
Subfolders Verbatim

Nothing special here, we specify the path of the directory (careful, the final / is important), and indicate that IMAP directories are stored in Maildir directories named the same way. At this point we can create the local directory:

mkdir -m 700 ~/mail

Finally we define the channel:

Channel backup
Far :gmail:
Near :local:
Patterns *
Create Near
Remove Near
Expunge Near

The Patterns instruction is used to indicate that we want to synchronize all IMAP directories. We also indicate that directory creation, directory removal and mail deletion is only performed on the local store.

To test that everything is correct, run mbsync -l backup. If everything is correct, mbsync will list all IMAP directories which will be synchronized.

To actually copy all emails, run mbsync --pull backup. Since I have a gigabit optic fiber connection, I was only limited by Google servers. Exporting more than 20 thousands messages (1.5GB) took roughly one hour and a half.

Systemd timer

At this point, you could simply run mbsync manually, but I am way too lazy for that. I use Archlinux, meaning that my init system is the controversial Systemd. While not everyone likes Systemd, it makes it easy to run periodic tasks.

First write the service file at ~/.config/systemd/user/mbsync.service:

[Unit]
Description=Backup emails
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/bin/mbsync --pull backup

[Install]
WantedBy=default.target

Then the timer itself at ~/.config/systemd/user/mbsync.timer:

[Unit]
Description=Backup emails

[Timer]
OnStartupSec=5min
OnUnitActiveSec=1h

[Install]
WantedBy=timers.target

We can now use systemctl to enable the service and the timer and execute it a first time:

systemctl --user enable mbsync.service
systemctl --user enable mbsync.timer
systemctl --user start mbsync

The local Maildir will now be updated 5 minutes after the user session starts, and every hour after that. You can check the status of the task with systemctl --user status mbsync.

Custom Font Lock configuration in Emacs

nicolas@n16f.net (Nicolas Martyanoff) — Fri, 24 Feb 2023 18:00:00 +0000

Font Lock is the builtin Emacs minor mode used to highlight textual elements in buffers. Major modes usually configure it to detect various syntaxic constructions and attach faces to them.

The reason I ended up deep into Font Lock is because I was not satisfied with the way it is configured for lisp-mode, the major mode used for both Common Lisp and Emacs Lisp code. This forced me to get acquainted with various aspects of Font Lock in order to change its configuration. If you want to change highlighting for your favourite major mode, you will find this article useful.

Common Lisp highlighting done wrong

The core issue of Common Lisp highlighting in Emacs is that a lot of it is arbitrary and inconsistent:

The mode highlights what it calls “definers” and “keywords”, but it does not really make sense in Common Lisp. Why would WITH-OUTPUT-TO-STRING be listed as a keyword, but not CLASS-OF?
SIGNAL uses font-lock-warning-face. Why would it be a warning? Even stranger, why would you use this warning face for CHECK-TYPE?
Keywords and uninterned symbols are all highlighted with font-lock-builtin-face. But they are not functions or variables. They are not even special in any way, and their syntax already indicates clearly their nature. Having so many yellow symbols everywhere is really distracting.
All symbols starting with & are highlighted using font-lock-type-face. But lambda list arguments are not types, and symbols starting with & are not always lambda list arguments.
All symbols preceded by ( whose name starts with DO- or WITH- are highlighted as keywords. There is even a comment by RMS stating that it is too general. He is right.

Beyond these issues, the mode sadly uses default Font Lock faces instead of defining semantically appropriate faces and mapping them to existing ones as default values.

The chances of successfully driving this kind of large and disruptive change directly into Emacs are incredibly low. Even if it was to be accepted, the result would not be available until the next release, which could mean months. Fortunately, Emacs is incredibly flexible and we can change all of this ourselves.

Note that you may not agree with the list of issues above, and this is fine. The point of this article is to show you how you can change the way Emacs highlights content in order to match your preferences. And you can do that for all major modes!

Font Lock configuration

Font Lock always felt a bit magic and it took me some time to find the motivation to read the documentation. As is turned out, it can be used for very complex highlighting schemes, but basic features are not that hard to use.

The main configuration of Font Lock is stored in the font-lock-defaults buffer-local variable. It is a simple list containing the following entries:

A list of symbols containing the value to use for font-lock-keywords at each level, the first symbol being the default value.
The value used for font-lock-keywords-only. If it is nil, it enables syntaxic highlighting (strings and comments) in addition of search-based (keywords) highlighting.
The value used for font-lock-keywords-case-fold-search. If true, highlighting is case insensitive.
The value used for font-lock-syntax-table, the association list controlling syntaxic highlighting. If it is nil, Font Lock uses the syntax table configured with set-syntax-table. In lisp-mode this would mean lisp-mode-syntax-table.
All remaining values are bindings using the form (VARIABLE-NAME . VALUE) used to set buffer-local values for other Font Lock variables.

The part we are interested about is search-based highlighting which uses regular expressions to find specific text fragments and attach faces to them.

Values used for font-lock-keywords are also lists. Each element is a construct used to specify one or more keywords to highlight. While these constructs can have multiple forms for more complex use cases, we will only use the two simplest ones:

(REGEXP . FACE) tells Font Lock to use FACE for text fragments which match REGEXP. For example, you could use ("\\_<-?[0-9]+\\_>" . font-lock-constant-face) to highlight integers as constants (note the use of \_< and \_> to match the start and end of a symbol; see the regexp documentation for more information).
(REGEXP (GROUP FACE)…) is a bit more advanced. When REGEXP matches a subset of the buffer, Font Lock assigns faces to the capture group identified by their number. You could use this construction to detect a complex syntaxic element and highlight some of its parts with different faces.

Simplified Common Lisp highlighting

We are going to configure keyword highlighting for the following types of values:

Character literals, e.g. #\Space.
Function names in the context of a function call for standard Common lisp functions.
Standard Common Lisp values such as *STANDARD-OUTPUT* or PI.

Additionally, we want to keep the default syntaxic highlighting configuration which recognizes character strings, documentation strings and comments.

Faces

Let us start by defining new faces for the different values we are going to match:

(defface g-cl-character-face
  '((default :inherit font-lock-constant-face))
  "The face used to highlight Common Lisp character literals.")

(defface g-cl-standard-function-face
  '((default :inherit font-lock-keyword-face))
  "The face used to highlight standard Common Lisp function symbols.")

(defface g-cl-standard-value-face
  '((default :inherit font-lock-variable-name-face))
  "The face used to highlight standard Common Lisp value symbols.")

Nothing complicated here, we simply inherit from default Font Lock faces. You can then configure these faces in your color theme without affecting other modes using Font Lock.

Keywords

To detect standard Common Lisp functions and values, we are going to need a regular expression. The first step is to build a list of strings for both functions and values. Easy to do with a bit of Common Lisp code!

(defun standard-symbol-names (predicate)
  (let ((symbols nil))
    (do-external-symbols (symbol :common-lisp)
      (when (funcall predicate symbol)
        (push (string-downcase (symbol-name symbol)) symbols)))
    (sort symbols #'string<)))
    
(standard-symbol-names #'fboundp)
(standard-symbol-names #'boundp)

The STANDARD-SYMBOL-NAMES build a list of symbols exported from the :COMMON-LISP package which satisfy a predicate. The first call gives us the name of all symbols bound to a function, and the second all which are bound to a value.

The astute reader will immediately wonder about symbols which are bound both a function and a value. They are easy to find by calling INTERSECTION on both sets of names: +, /, *, -. It is not really a problem: we can highlight function calls by matching function names preceded by (, making sure that these symbols will be correctly identified as either function symbols or value symbols depending on the context.

We store these lists of strings in the g-cl-function-names and g-cl-value-names (the associated code is not reproduced here: these lists are quite long; but I posted them as a Gist).

With this lists, we can use the regexp-opt Emacs Lisp function to build optimized regular expressions matching them:

(defvar g-cl-font-lock-keywords
  (let* ((character-re (concat "#\\\\" lisp-mode-symbol-regexp "\\_>"))
         (function-re (concat "(" (regexp-opt g-cl-function-names t) "\\_>"))
         (value-re (regexp-opt g-cl-value-names 'symbols)))
    `((,character-re . 'g-cl-character-face)
      (,function-re
       (1 'g-cl-standard-function-face))
      (,value-re . 'g-cl-standard-value-face))))

Characters literals are reasonably easy to match.

Functions are a bit more complicated since we want to match the function name when it is preceded by an opening parenthesis. We use a capture capture (see the last argument of regexp-opt) for the function name and highlight it separately.

Values are always matched as full symbols: we do not want to highlight parts of a symbol, for example MAP in a symbol named MAPPING.

Final configuration

Finally we can define the variable which will be used for font-lock-defaults in the initialization hook; we copy the original value from lisp-mode, and change the keyword list for what is going to be our own configuration:

(defvar g-cl-font-lock-defaults
  '((g-cl-font-lock-keywords)
    nil                                 ; enable syntaxic highlighting
    t                                   ; case insensitive highlighting
    nil                                 ; use the lisp-mode syntax table
    (font-lock-mark-block-function . mark-defun)
    (font-lock-extra-managed-props help-echo)
	(font-lock-syntactic-face-function
	 . lisp-font-lock-syntactic-face-function)))

To configure font-lock-defaults, we simply set it in the initialization hook of lisp-mode:

(defun g-init-lisp-font-lock ()
  (setq font-lock-defaults g-cl-font-lock-defaults))
  
(add-hook 'lisp-mode-hook 'g-init-lisp-font-lock)

Comparison

Let us compare highlighting for a fragment of code before and after our changes:

The differences are subtle but important:

All standard functions are highlighted, helping to distinguish them from user-defined functions.
Standard values such as *ERROR-OUTPUT* are highlighted.
Character literals are highlighted the same way as character strings.
Keywords are not highlighted anymore, avoiding the confusion with function names.

Conclusion

That was not easy; but as always, the effort of going through the documentation and experimenting with different Emacs components was very rewarding. Font Lock does not feel like a black box anymore, opening the road for the customization of other major modes.

In the future, I will work on a custom color scheme to use more subtle colors, with the hope of reducing the rainbow effect of so many major modes, including lisp-mode.

Common Lisp implementations in 2023

nicolas@n16f.net (Nicolas Martyanoff) — Wed, 22 Feb 2023 18:00:00 +0000

Much has been written on Common Lisp; there is rarely one year without someone proclaming the death of the language and how nobody uses it anymore. And yet it is still here, so something must have been done right.

Common Lisp is not a software, it is a language described by the ANSI INCITS 226-1994 standard; there are multiple implementations available, something often used as argument for how alive and thriving the language is.

Let us see what the 2023 situation is.

General information

Implementation	License	Target	Last release
SBCL	Public domain	Native	2023/01 (2.3.1)
CCL	Apache 2.0	Native	2021/05 (1.12.1)
ECL	LGPL 2.1	Native (C translation)	2021/02 (21.2.1)
ABCL	GPL2	Java bytecode	2023/02 (1.9.1)
CLASP	LGPL 2.1	Native (LLVM)	2023/01 (2.1.0)
CMUCL	Public domain	Native	2017/10 (21c)
GCL	LGPL2	Native (C translation)	2023/01 (2.6.14)
CLISP	GPL	Bytecode	2010/07 (2.49)
Lispworks	Proprietary	Native	2022/06 (8.0.1)
Allegro	Proprietary	Native	2017/04 (10.1)

Note that all projects may have small parts with different licenses. This is particularily important for CLASP which contains multiple components imported from other projects.

I was quite surprised to see so many projects with recent releases. Clearly a good sign. Let us look at each implementation.

Implementations

SBCL

Steel Bank Common Lisp was forked from CMUCL in December 1999 and has since massively grown in popularity; it is currently the most used implementation by far. Unsurprisingly given its popularity, SBCL is supported by pretty much all Common Lisp libraries and tools out there. It is well known for generating fast native code compared to other implementations.

The most important aspect of SBCL is that it is actively maintained: its developers release new versions on a monthly basis, bringing each time a small list of improvements and bug fixes. Activity has actually increased these last years, something uncommon in the Common Lisp world.

CCL

Clozure Common Lisp has a long and complex history and has been around for decades. It is a mature implementation; it has two interesting aspects compared to SBCL:

The compiler is much faster.
Error messages tend to be clearer.

This is why I currently use it to test my code along SBCL. And according to what I have heard, this is a common choice among developers.

The main issue with CCL is that the project is almost completely abandonned. Git activity has slowed down to a crawl in the last two years, and none of the original maintainers from Clozure seem to be actively working on it. It remains nonetheless a major implementation.

ECL

Embeddable Common Lisp is a small implementation which can be used both as a library or as a standalone program. It contains a bytecode interpreter, but can also translate Lisp code to C to be compiled to native code.

While development is slow, improvements and bug fixes are still added on a regular basis. Clearly an interesting project: I could see myself using ECL to write plugins into an application able to call a C library.

ABCL

Armed Bear Common Lisp is quite different from other implementations: it produces Java bytecode and targets the Java Virtual Machine, making it a useful tool in Java ecosystems.

While it has not found the same success as Clojure, ABCL is still a fully featured Common Lisp implementation which passes almost the entire ANSI Common Lisp test suite.

Developement is slow nowadays but there are still new releases with lots of bug fixes. Also note that two of the developers are able to provide paid support.

CLASP

CLASP is a newcomer in the Common Lisp world (new meaning it is less than a decade old). Developed by Christian Schafmeister for his research work, this implementation has been used as an exemple of how alive and kicking Common Lisp, mainly due to two excellent presentations.

While very promising, CLASP suffers from its young age: trying to run the last release on my code resulted in a brutal error with no details and no backtrace. However I have no doubt that CLASP will get a lot better: it is actively maintained and used in production, two of the necessary ingredients for a software to stay relevant.

GCL

GNU Common Lisp is described as the official Common Lisp implementation for the GNU project. While it clearly does not have the popularity of other implementations, it is still a maintained project.

Trying to use it, I quickly realized it is not fully compliant with the standard. For example it will fail when evaluating a call to COMPILE-FILE with the :VERBOSE key argument.

Hopefully development will continue.

CLISP

CLISP is almost as old as I am; it was the first implementation I used a long time ago, and it still works. While it has all the usual features (multithreading, FFI, MOP, etc.), there is no real reason to use it compared to other implementations.

Even if it was to have any specific feature, CLISP is almost completely abandonned. While there are has been a semblant of activity a few years ago, active development pretty much stopped around 2012; the last release was more than 12 years ago.

Lispworks

Moving to proprietary implementations; Lispworks has been around for more than 30 years and the company producing it still release new versions on a regular basis.

While Lispworks supports most features you would expect from a commercial product (native compiler, multithreading, FFI, GUI library, various graphical tools, a Prolog implementation…), it is hampered by its licensing system.

The free “Personal Edition” limits the program size and the amount of time it can run, making it pretty much useless for anything but evaluation. The professional and enterprise licenses do not really make sense for anyone: you will have to buy separate licenses for every single platform at more than a thousand euros per license (with the enterprise version being 2-3 times more expensive). Of course you will have to buy a maintenance contract on a yearly basis… but it does not include technical support. It will have to be bought with “incident packs” costing thousands of euros; because yes, paying for a product and a maintenance contract does not mean they will fix bugs, and you will have to pay for each of them.

I do not have anything personal against commercial software, and I strongly support developers being paid for their work. But this kind of licensing makes Lispworks irrelevant to everyone but those already using their proprietary libraries.

Allegro

Allegro Common Lisp is the other well known proprietary implementation. Developped by Franz Inc., it is apparently used by multiple organizations including the U.S. Department of Defense.

Releases are uncommon, the last one being almost 6 years ago. But Allegro is a mature implementation packed with features not easily replicated such as AllegroCache, AllegroServe, libraries for multiple protocols and data formats, analysis tools, a concurrent garbage collector and even an OpenGL interface.

Allegro suffers the same issue as Lispworks: the enterprise-style pricing system is incredibly frustrating. The website advertises a hefty $599 starting price (which at least includes technical support), but there is no mention of what it contains. Interested developpers will have to contact Franz Inc. to get other prices. A quick Google search will reveal rumours of enterprise versions priced above 8000 dollars. No comment.

Conclusion

Researching Common Lisp implementations has been interesting. While it is clear that the language is far from dead, its situation is very fragile. Proprietary implementations are completely out of touch with the needs of most developers, leaving us with a single open source, actively maintained, high performance implementation: SBCL. Unless of course they are willing to deal with the JVM to use ABCL.

It might me interesting to investigate a possible solution to keep CCL somehow alive, with patches being merged and releases being produced. I sent a patch very recently, let us see what can be done!

Reading files faster in Common Lisp

nicolas@n16f.net (Nicolas Martyanoff) — Wed, 15 Feb 2023 18:00:00 +0000

While Common Lisp has functions to open, read and write files, none of them takes care of reading and returning the entire content. This is something that I do very regularly, so it made sense to add such a function to Tungsten. It turned out to be a bit more complicated than expected.

A simple but incorrect implementation

The simplest implementation relies on the FILE-LENGTH function which returns the length of a stream (which of course only makes sense for a file stream). The Hyperspec clearly states that “for a binary file, the length is measured in units of the element type of the stream”. Since we are only reading binary data, everything is fine.

Let us write the function:

(defun read-file (path)
  (declare (type (or pathname string) path))
  (with-open-file (file path :element-type 'core:octet)
    (let ((data (make-array (file-length file) :element-type 'core:octet)))
      (read-sequence data file)
      data)))

Note that CORE:OCTET is a Tungsten type for (UNSIGNED-BYTE 8).

The function works as expected, returning the content of the file as an octet vector. But it is not entirely correct.

This implementation only works for regular files. Various files on UNIX will report a length of zero but can still be read. Now you might protest that it would not make sense to call READ-FILE on a device such as /dev/urandom, and you would be right. But a valid example would be pseudo files such as those part of procfs. If you want to obtain memory stats about your process on Linux, you can simply read /proc/self/statm. But this is not a regular file and READ-FILE will return an empty octet vector.

Doing it right and slow

The right way to read a file is to read its content block by block until the read operation fails because it reached the end of the file.

Let us re-write READ-FILE:

(defun read-file (path)
  (declare (type (or pathname string) path))
  (let ((data (make-array 0 :element-type 'core:octet :adjustable t))
        (block-size 4096)
        (offset 0))
    (with-open-file (file path :element-type 'core:octet)
      (loop
        (let* ((capacity (array-total-size data))
               (nb-left (- capacity offset)))
          (when (< nb-left block-size)
            (let ((new-length (+ capacity (- block-size nb-left))))
              (setf data (adjust-array data new-length)))))
        (let ((end (read-sequence data file :start offset)))
          (when (= end offset)
            (return-from read-file (adjust-array data end)))
          (setf offset end))))))

This time we rely on an adjustable array; we iterate, making sure we have enough space in the array to read an entire block each time. When the array is too short, we use ADJUST-ARRAY to extend it, relying on its ability to reuse the underlying storage instead of systematically copying its content.

Finally, once READ-SEQUENCE stops returning data, we truncate the array to the right size and return it.

This function worked correctly and I started using it regularly. Recently I started working with a file larger than usual and realized that READ-FILE was way too slow. With a NVMe drive, I would expect to be able to read a 10+MB file almost instantaneously, but it took several seconds.

After inspecting the code to find what could be so slow, I started to wonder about ADJUST-ARRAY; while I thought SBCL would internally extend the underlying memory in large blocks to minimize allocations, behaving similarly to realloc() in C, it turned out not to be the case. While reading the code behind ADJUST-ARRAY, I learned that it precisely allocates the required size. As a result, this implementation of READ-FILE performs one memory allocation for each 4kB block. Not a problem for small files, slow for larger ones.

A final version, correct and fast

Since I understood what the problem was, fixing it was trivial. When there is not enough space to read a block, we extend the array by at least 50% of its current size. Of course this is a balancing act: for example doubling the size at each allocation would reduce even more the number of allocations, but would increase the total amount of memory allocated. The choice is up to you.

(defun read-file (path)
  (declare (type (or pathname string) path))
  (let ((data (make-array 0 :element-type 'core:octet :adjustable t))
        (block-size 4096)
        (offset 0))
    (with-open-file (file path :element-type 'core:octet)
      (loop
        (let* ((capacity (array-total-size data))
               (nb-left (- capacity offset)))
          (when (< nb-left block-size)
            (let ((new-length (max (+ capacity (- block-size nb-left))
                                   (floor (* capacity 3) 2))))
              (setf data (adjust-array data new-length)))))
        (let ((end (read-sequence data file :start offset)))
          (when (= end offset)
            (return-from read-file (adjust-array data end)))
          (setf offset end))))))

This last version reads a 250MB file in a quarter of a second, while the original version took almost two minutes. Much better!

Templating in Emacs with Tempo

nicolas@n16f.net (Nicolas Martyanoff) — Sun, 12 Feb 2023 18:00:00 +0000

I have used text editors for more than 20 years; in all that time, I have never used a templating system to generate content because copy pasting and replacing always felt good enough. I recently decided to give it a try.

In Emacs it is hard to talk about templating without mentionning YASnippet. Developped by the prolific João Távora, who also developped Eglot, YASnippet lets you write templates as text documents.

But while I was researching the subject, I found out that Emacs already had two builtin templating modules: Skeleton and Tempo.

Tempo looked simpler, so I decided to give it a chance.

Using Tempo

Tempo templates are functions defined with tempo-define-template. The content to be inserted is a S-expression containing various kinds of elements which control the insertion process.

As an example, let us define a template to insert a HTML figure.

First we define a variable to store a list of templates. When using html-mode, we instruct Tempo to use it.

(defvar g-html-tempo-tags nil)

(defun g-init-html-tempo-templates ()
  (tempo-use-tag-list 'g-html-tempo-tags))

(add-hook 'html-mode-hook 'g-init-html-tempo-templates)

Then we define the template itself:

(tempo-define-template
 "g-html-figure"
 '("<figure>" n
   > "<img src=\"" (p "URI: ") "\">" n
   > "<figcaption>" (p "Caption: ") "</figcaption>" n
   "</figure>")
 "figure"
 "a figure containing an image and caption"
 'g-html-tempo-tags)

This form creates a function named tempo-template-g-html-figure. When it is called, Tempo processes elements of the template:

Strings are inserted in the buffer.
The n symbol causes the insertion of a new line character
The > symbol tells Emacs to indent the current line according to the rules of the major mode of the buffer.
The p forms cause Tempo to ask the user for values to insert. Note that you need to set tempo-interative to t.

Note that Tempo supports more elements; refer to the documentation string of tempo-define-template for more information.

The template is associated to the figure text tag which will be used for completion. We also add a description, and finally add the template to the g-html-tempo-tags list. Note that these three last arguments are optional.

Tag matching

Of course we do not have to call the template manually. When we call tempo-complete-tag, Tempo uses the string before the cursor to decide which template to insert. Open a HTML buffer, type figure and execute tempo-complete-tag (I bind it to M-S-<tab>): Tempo will automatically insert our template.

Tempo will handle the case where there is partial match for multiple templates and will spawn a completion buffer.

Now it would make sense to use <figure> as tag. To do so, update the g-init-html-tempo-templates function to set the local variable that Tempo uses to detect a tag:

(setq tempo-match-finder "\\(<[a-z]+>\\)\\=")

Doing so will use HTML tags as Tempo tags. We can then alter the call to tempo-define-template to use <figure> as tag, and from now on use it for template insertion.

Of course this means we can customize it to allow different formats of Tempo tags depending on the major mode we are in. Handy.

Writing a template selector

Calling template functions manually is unpractical and tag completion requires remembering which tags have been defined. My interface of choice would be a simple key spawning an incremental completion buffer (Helm in my case) to let me select a template to insert.

As it turns out, it is not that hard:

(defun g-insert-tempo-template ()
  (interactive)
  (let* ((tags-data
          (mapcar (lambda (entry)
                    (let ((function (cdr entry)))
                      (list function (documentation function))))
                  (tempo-build-collection)))
         (completion-extra-properties
          `(:annotation-function
            (lambda (string)
              (let* ((data (alist-get string minibuffer-completion-table
                                      nil nil #'string=))
                     (description (car data)))
                (format "  %s" description)))))
         (function-name (completing-read "Template: " tags-data))
         (function (intern function-name)))
    (funcall function)))

As we have already seen in a previous post, completing-read is quite limited in terms of presentation. I will probably spend some time switching from Helm to a mix of Vertico and Marginalia which apparently offers more options.

This will do the job in the mean time.

Conclusion

Tempo is reasonably satisfying. While it is a very simple module, it lets me define templates as Emacs Lisp expressions, associate them to tags and store them in tag lists which can be used in the major modes of my choice.

I do not expect to start using dozens of tiny templates for the simplest constructions, but Tempo is going to help with recurrent complex constructions such as Emacs module skeletons or Common Lisp system definitions.

Decluttering Dired for peace of mind

nicolas@n16f.net (Nicolas Martyanoff) — Fri, 03 Feb 2023 18:00:00 +0000

Dired is an Emacs mode providing an interactive interface to navigate and manipulate files. It took me a while to start using it, but it is so efficient that I now spend more time in it than in my usual shell.

However I am no fond of the way files are listed. Dired displays a lot of information which is — at least for me — rarely used. As a result the important content (the name of the file) is flushed to the right of the window, often leading to unpractical line wrapping.

It is hard to find what you are searching for in this kind of interface:

So I decided to spend some time improving my Dired experience.

Configuring a simpler view

Dired comes with the dired-hide-details-mode minor mode which hides extra information. By default it is activated with the ( key.

It has two drawbacks:

First it also hides the display of symbolic link target while still showing the -> marker after the link. This behaviour can be changed by setting the dired-hide-details-hide-symlink-targets variable to nil.
Second, Dired disables this minor mode each time you move to a different directory, which is really annoying

We create a g-dired-minimal-view variable to store the current state. When then write two functions: one to either enable or disable the minor mode depending on the state, called by the dired-mode-hook, and one to switch between simple and extended view bound to the more accessible <tab> key.

(setq g-dired-minimal-view t)

(defun g-dired-setup-view ()
  (dired-hide-details-mode (if g-dired-minimal-view 1 -1)))

(defun g-dired-switch-view ()
  (interactive)
  (setq g-dired-minimal-view (not g-dired-minimal-view))
  (g-dired-setup-view))

(use-package dired
  :config
  (setq dired-hide-details-hide-symlink-targets nil)

  :hook
  ((dired-mode-hook . g-dired-setup-view))

  :bind
  (:map dired-mode-map
        ("<tab>" . g-dired-switch-view)))

A better extended view

While I prefer the simple view by default, the extended view still has its use; but I find it hard to read:

The traditional representations of permissions takes a lot of horizontal space, the 4 digit octal value would be shorter.
The number of links is not really useful.
The size of the file would be more convenient in a human readable format.
The date and time uses an irregular US-style format which is either “month day year” or “month day hour:minute”.

To list files, Dired call the ls program with a list of options (defined by the dired-listing-switches variable), insert the output in the buffer and rely on regular expressions to extract information. This means that we have very little control on formatting. We can change the options passed to ls, but we have to be careful not to break Dired features. For example, Dired can highlight files with specific permissions, but rely on a regular expression based on the format used by ls, so we cannot change the permission column.

First we use dired-listing-switches to add the -h (for human-readable sizes) and --time-style=long-iso option. Since these options are only available for the GNU version of ls, I will have to add platform detection for FreeBSD support in a way that works when Dired is running with TRAMP. Something for another day.

Then we add a dired-after-readin-hook hook which is called by Dired after the output of ls has been collected and inserted in the buffer. In this hook, we iterate on lines and process them. We highlight metadata with the shadow face to keep them visually distinct from the filename, and we hide the link count column using the invisible text property so that the text stays in the buffer, making sure not to break Dired features.

Finally we disable wrapping with a hook.

(defun g-dired-postprocess-ls-output ()
  "Postprocess the list of files printed by the ls program when
executed by Dired."
  (save-excursion
    (goto-char (point-min))
    (while (not (eobp))
      ;; Go to the beginning of the next line representing a file
      (while (null (dired-get-filename nil t))
        (dired-next-line 1))
      (beginning-of-line)
      ;; Narrow to the line and process it
      (let ((start (line-beginning-position))
            (end (line-end-position)))
        (save-restriction
          (narrow-to-region start end)
          (setq inhibit-read-only t)
          (unwind-protect
              (g-dired-postprocess-ls-line)
            (setq inhibit-read-only nil))))
      ;; Next line
      (dired-next-line 1))))

(defun g-dired-disable-line-wrapping ()
  (setq truncate-lines t))

(defun g-dired-postprocess-ls-line ()
  "Postprocess a single line in the ls output, i.e. the information
about a single file. This function is called with the buffer
narrowed to the line."
  ;; Highlight everything but the filename
  (when (re-search-forward directory-listing-before-filename-regexp nil t 1)
    (add-text-properties (point-min) (match-end 0) '(font-lock-face shadow)))
  ;; Hide the link count
  (beginning-of-line)
  (when (re-search-forward " +[0-9]+" nil t 1)
    (add-text-properties (match-beginning 0) (match-end 0) '(invisible t))))

(use-package dired
  :config
  (setq dired-listing-switches "-alh --time-style=long-iso")

  :hook
  ((dired-mode-hook . g-dired-disable-line-wrapping)
   (dired-after-readin-hook . g-dired-postprocess-ls-output)))

Much better now!

Using units in Emacs Calc

nicolas@n16f.net (Nicolas Martyanoff) — Sat, 28 Jan 2023 18:00:00 +0000

I was recently reminded of the ability of Emacs Calc to perform computations on values with units. Like so many Emacs features, it was on the list of the things I wanted to explore. Today is the day to do so!

Basic operations with units

The Emacs Calc manual presents a lot of features but makes it hard to understand how to start. You cannot just enter an expression such as “25mm” in calc-mode: 25 will be correctly interpreted as a number, but “mm” will be used to trigger the m m key binding which executes calc-save-modes. Not what you had in mind.

To enter a value with a unit, use the ' (apostrophe) key to enter an algebraic expression and hit return to push it on the stack. You can now use this value for your computations.

Note that after hitting ', you can use it a second time to recall the expression on the stack and edit it.

To remove the unit associated with a value. Simply hit u r, for “unit remove”.

Simplifying expressions

Expressions containing multiple units can be simplified. For example, enter two values, 2m and 50cm and add them with +. The resulting expression is 2 m + 50 cm. Hit u s, for “unit simplify”, to have Calc simplify it to 2.5 m.

Converting units

Unit conversion is of course possible. Use u c, for “unit convert”, at any moment: Calc will ask for a unit and will try to convert the value at the top of the stack into this unit. For example, entering 1.5mi (mi being the unit for miles) and typing u c km <return> will correctly yield 2.414016 km.

You will quickly discover that Calc tries way too hard to convert units and will produce really strange results when types are not compatible. For example, trying to convert 250mA into kilometers will produce 0.25A which is clearly incorrect. I have no idea what causes this behaviour. Fortunately you can use u n, bound to calc-convert-exact-units, which will check that the unit you required is compatible with the unit of the expression on the stack. To minimize the risk of producing incorrect results by mistake, I replace the calc-convert-units function by calc-convert-exact-units. I would prefer to remap u c, but Calc has a strange way to handle key bindings that prevent doing this kind of remapping.

(use-package calc
  :config
  (require 'calc-units)

  (setf (symbol-function 'calc-convert-units)
        (symbol-function 'calc-convert-exact-units)))

As a shortcut, u b, where b stands for “base”, will convert the value to its base unit. For example, it will automatically convert 250mA to 0.25A.

Defining custom units

Calc comes by default with more than 170 units (type u v to list them all), but none of them are about standard data size units. This is disappointing given that Emacs is a software primarily used for programming. But we can add new units, so it is not that much of a problem.

New units can be defined with the math-additional-units variable. Its format is the same as math-standard-units: each entry is a list of three elements:

A symbol identifying the unit.
An expression indicating the value of the unit, or nil for fundamental units.
A textual description.

Let us add data sizes. We need two units, bits and bytes, with bytes being defined as 8 bits. Note that using the b symbol for the bit unit will override the internal “Barn” unit; not a problem to me, I do not expect to use it any time soon. Feel free to use bit instead.

(setq math-additional-units
      '((b nil "Bit")
        (B "8 * b" "Byte")))

(setq math-units-table nil)

We need to set math-units-table to nil after defining new units to force Calc to recompute its internal table.

These new units can of course be used with standard SI prefixes defined in Calc. For example, 25kB will correctly be converted to 25000 B. And asking for a convertion to the base unit will yield 200000 b.

A limitation of Calc is its inability to handle unit prefixes of more than one character, so we cannot define ki as being a prefix with a multiplicative value of 1024, which would for example allow us to manipulate kibibytes.

A workaround is to define these power-of-two binary units with the prefix included:

(setq math-additional-units
      '((b nil "Bit")
        (B "8 * b" "Byte")

        (kiB "2^10 * B" "Kibibyte")
        (MiB "2^20 * B" "Mebibyte")
        (GiB "2^30 * B" "Gibibyte")
        (TiB "2^40 * B" "Tebibyte")
        (PiB "2^50 * B" "Pebibyte")
        (EiB "2^60 * B" "Exbibyte")))

(setq math-units-table nil)

Very helpful for data size conversion.

Going farther with Calc

Writing this post has been an interesting experience: it forced me to read various parts of the Calc manual and some of its source files. It made me realize that this package is full of useful features. I hope to find the time to experiment with Calc features in the future.

Custom Common Lisp indentation in Emacs

nicolas@n16f.net (Nicolas Martyanoff) — Mon, 23 Jan 2023 18:00:00 +0000

While SLIME is most of the time able to indent Common Lisp correctly, it will sometimes trip on custom forms. Let us see how we can customize indentation.

In the process of writing my PostgreSQL client in Common Lisp, I wrote a READ-MESSAGE-CASE macro which reads a message from a stream and execute code depending on the type of the message:

(defmacro read-message-case ((message stream) &rest forms)
  `(let ((,message (read-message ,stream)))
     (case (car ,message)
       (:error-response
        (backend-error (cdr ,message)))
       (:notice-response
        nil)
       ,@forms
       (t
        (error 'unexpected-message :message ,message)))))

This macro is quite useful: all message loops can use it to automatically handle error responses, notices, and signal unexpected messages.

But SLIME does not know how to indent READ-MESSAGE-CASE, so by default it will align all message forms on the first argument:

(read-message-case (message stream)
                   (:authentication-ok
                     (return))
                   (:authentication-cleartext-password
                     (unless password
                       (error 'missing-password))
                     (write-password-message password stream)))

While we want it aligned the same way as HANDLER-CASE:

(read-message-case (message stream)
  (:authentication-ok
    (return))
  (:authentication-cleartext-password
    (unless password
      (error 'missing-password))
    (write-password-message password stream)))

Good news, SLIME indentation is defined as a list of rules. Each rule associates an indentation specification (a S-expression describing how to indent the form) to a symbol and store it as the common-lisp-indent-function property of the symbol.

You can obtain the indentation rule of a Common Lisp symbol easily. For example, executing (get 'defun 'common-lisp-indent-function) (e.g. in IELM or with eval-expression) yields (4 &lambda &body). This indicates that DEFUN forms are to be indented as follows:

The first argument of DEFUN (the function name) is indented by four spaces.
The second argument (the list of function arguments) is indented as a lambda list.
The rest of the arguments are indented based on the lisp-body-indent custom variable, which controls the indentation of the body of a lambda form (two spaces by default).

You can refer to the documentation of the common-lisp-indent-function Emacs function (defined in SLIME of course) for a complete description of the format.

We want READ-MESSAGE-CASE to be indented the same way as HANDLER-CASE, whose indentation specification is (4 &rest (&whole 2 &lambda &body)) (in short, an argument and a list of lambda lists). Fortunately there is a way to specify that a form must be indented the same way as another form, using (as <symbol>).

Let us first define a function to set the indentation specification of a symbol:

(defun g-common-lisp-indent (symbol indent)
  "Set the indentation of SYMBOL to INDENT."
  (put symbol 'common-lisp-indent-function indent))

Then use it for READ-MESSAGE-CASE:

(g-common-lisp-indent 'read-message-case '(as handler-case))

While it is in general best to avoid custom indentation, exceptions are sometimes necessary for readability. And SLIME makes it easy.

Taking code screenshots in Emacs

nicolas@n16f.net (Nicolas Martyanoff) — Wed, 18 Jan 2023 18:00:00 +0000

My friend Rémi found an Emacs package developed by Tecosaur to take screenshots of your code. I usually do it with scrot, but it always requires spending some time cropping the result with Gimp. A bit annoying.

Installation

Let us give it a try. The screenshot package is not available on MELPA due to cosmetic issues, so we have two options.

The first one would be to clone the repository, add its location to load-path, and use require to load the module. But doing so means having to explicitely install its dependencies, transient and posframe.

The second option is to rely on use-package and straight.el. With straight.el, you can load packages from various sources, including Git repositories.

(use-package screenshot
  :straight (:type git :host github :repo "tecosaur/screenshot"))

Evaluate the form with C-x C-e, and straight.el will clone the repository and load the screenshot package.

Usage

Simply run M-x screenshot to take a screenshot. If you are currently selecting a region, it will screenshot this part only. If not, it will use the entire buffer.

Since Screenshot uses the transient library, it has the same well designed interface system as Magit:

I really like the ability to adjust the font size just for the screenshot. By default, the save action will save the image in the same directory as the buffer which was selected when screenshot was executed.

Customization

Screenshot lets you change the default value of each setting. This is not obvious when reading the code, because the variables used are defined programmatically through two layers of macros: first screenshot--define-infix then transient-define-infix.

It is too bad these values are not defined as proper settings using defcustom, but we can still set them.

This is a list of available settings:

Variable	Description
`screenshot-line-numbers-p`	Whether to display line numbers or not.
`screenshot-relative-line-numbers-p`	Whether to use relative line numbers or not.
`screenshot-text-only-p`	Whether to show code as simple text or not.
`screenshot-truncate-lines-p`	Whether to truncate long lines instead of wrapping them.
`screenshot-font-family`	The name of the font to use.
`screenshot-font-size`	The size of the font to use.
`screenshot-border-width`	The width of the border.
`screenshot-radius`	The corner radius of the border.
`screenshot-min-width`	The minimal width of the screenshot as a number of characters.
`screenshot-max-width`	The maximal width of the screenshot as a number of characters.
`screenshot-shadow-radius`	The corner radius of the shadow.
`screenshot-shadow-intensity`	The intensity of the shadow.
`screenshot-shadow-color`	The hexadecimal code of the shadow.
`screenshot-shadow-offset-horizontal`	The horizontal offset of the shadow in pixels.
`screenshot-shadow-offset-vertical`	The vertical offset of the shadow in pixels.

Another nice feature is the presence of a hook executed in the temporary buffer used for the screenshot, allowing to execute code affecting this buffer just before the actual screenshot is being taken.

In my case, I use it to disable the fill column indicator and to remove the additional line spacing added by default by Screenshot.

This is my configuration:

(defun g-screenshot-on-buffer-creation ()
  (setq display-fill-column-indicator-column nil)
  (setq line-spacing nil))

(use-package screenshot
  :straight (:type git :host github :repo "tecosaur/screenshot")

  :config
  (setq screenshot-line-numbers-p nil)

  (setq screenshot-min-width 80)
  (setq screenshot-max-width 80)
  (setq screenshot-truncate-lines-p nil)

  (setq screenshot-text-only-p nil)

  (setq screenshot-font-family "Berkeley Mono")
  (setq screenshot-font-size 10)

  (setq screenshot-border-width 16)
  (setq screenshot-radius 0)

  (setq screenshot-shadow-radius 0)
  (setq screenshot-shadow-offset-horizontal 0)
  (setq screenshot-shadow-offset-vertical 0)

  :hook
  ((screenshot-buffer-creation-hook . g-screenshot-on-buffer-creation)))

All in all a very useful package!

ANSI color rendering in SLIME

nicolas@n16f.net (Nicolas Martyanoff) — Mon, 16 Jan 2023 18:00:00 +0000

I was working on the terminal output for a Common Lisp logger, and I realized that SLIME does not interpret ANSI escape sequences.

This is not the end of the world, but having at least colors would be nice. Fortunately there is a library to do just that.

First let us install the package, here using use-package and straight.el.

(use-package slime-repl-ansi-color
  :straight t)

While in theory we are supposed to just add slime-repl-ansi-color to slime-contribs, it did not work for me, and I add to enable the minor mode manually.

If you already have a SLIME REPL hook, simply add (slime-repl-ansi-color-mode 1). If not, write an initialization function, and add it to the SLIME REPL initialization hook:

(defun g-init-slime-repl-mode ()
  (slime-repl-ansi-color-mode 1))
  
(add-hook 'slime-repl-mode-hook 'g-init-slime-repl-mode)

To test that it works as intended, fire up SLIME and print a simple message using ANSI escape sequences:

(let ((escape (code-char 27)))
  (format t "~C[1;33mHello world!~C[0m~%" escape escape))

While it is tempting to use the #\Esc character, it is part of the Common Lisp standard; therefore we use CODE-CHAR to obtain it from its ASCII numeric value. We use two escape sequences, the first one to set the bold flag and foreground color, and the second one to reset display status.

If everything works well, should you see a nice bold yellow message:

Switching between implementations with SLIME

nicolas@n16f.net (Nicolas Martyanoff) — Thu, 12 Jan 2023 18:00:00 +0000

While I mostly use SBCL for Common Lisp development, I regularly switch to CCL or even ECL to run tests.

This is how I do it with SLIME.

Starting implementations

SLIME lets you configure multiple implementations using the slime-lisp-implementations setting. In my case:

(setq slime-lisp-implementations
   '((sbcl ("/usr/bin/sbcl" "--dynamic-space-size" "2048"))
     (ccl ("/usr/bin/ccl"))
     (ecl ("/usr/bin/ecl"))))

Doing so means that running M-x slime will execute the first implementation, i.e. SBCL. There are two ways to run other implementations.

First you can run C-u M-x slime which lets you type the path and arguments of the implementation to execute. This is a bit annoying because the prompt starts with the content of the inferior-lisp-program variable, i.e. "lisp" by default, meaning it has to be deleted manually each time. Therefore I set inferior-lisp-program to the empty string:

(setq inferior-lisp-program "")

Then you can run C-- M-x slime (or M-- M-x slime which is easier to type) to instruct SLIME to use interactive completion (via completing-read) to let you select the implementations among those configured in slime-lisp-implementations.

To make my life easier, I bind C-c C-s s to a function which always prompt for the implementation to start:

(defun g-slime-start ()
  (interactive)
  (let ((current-prefix-arg '-))
    (call-interactively 'slime)))

Using C-c C-s as prefix for all my global SLIME key bindings helps me remember them.

Switching between multiple implementations

Running the slime function several times will create multiple connections as expected. Commands executed in Common Lisp buffers are applied to the current connection, which is by default the most recent one.

There are two ways to change the current implementation:

Run M-x slime-next-connection.
Run M-x slime-list-connections, which opens a buffer listing connections, and lets you choose the current one with the d key.

I find both impractical: the first one does not let me choose the implementation, forcing me to run potentially several times before getting the one I want. The second one opens a buffer but does not switch to it.

All I want is a prompt with completion. So I wrote one.

First we define a function to select a connection among existing one:

(defun g-slime-select-connection (prompt)
  (interactive)
  (let* ((connections-data
          (mapcar (lambda (process)
                    (cons (slime-connection-name process) process))
                  slime-net-processes))
         (completion-extra-properties
          '(:annotation-function
            (lambda (string)
              (let* ((process (alist-get string minibuffer-completion-table
                                         nil nil #'string=))
                     (contact (process-contact process)))
                (if (consp contact)
                    (format "  %s:%s" (car contact) (cadr contact))
                  (format "  %S" contact))))))
         (connection-name (completing-read prompt connections-data)))
    (let ((connection (cl-find connection-name slime-net-processes
                               :key #'slime-connection-name
                               :test #'string=)))
      (or connection
          (error "Unknown SLIME connection %S" connection-name)))))

Then use it to select a connection as the current one:

(defun g-slime-switch-connection ()
  (interactive)
  (let ((connection (g-slime-select-connection "Switch to connection: ")))
    (slime-select-connection connection)
    (message "Using connection %s" (slime-connection-name connection))))

I bind this function to C-c C-s c.

In a perfect world, we could format nice columns in the prompt and highlight the current connection, but the completing-read interface is really limited, and I did not want to use an external package such as Helm.

Stopping implementations

Sometimes it is necessary to stop an implementations and kill all associated buffers. It is not something I use a lot; but when I need it, it is frustrating to have to switch to the REPL buffer, run slime-quit-lisp, then kill the REPL buffer manually.

Adding this feature is trivial with the g-slime-select-connection defined earlier:

(defun g-slime-kill-connection ()
  (interactive)
  (let* ((connection (g-slime-select-connection "Kill connection: "))
         (repl-buffer (slime-repl-buffer nil connection)))
    (when repl-buffer
      (kill-buffer repl-buffer))
    (slime-quit-lisp-internal connection 'slime-quit-sentinel t)))

Finally I bind this function to C-c C-s k.

It is now much more comfortable to manage multiple implementations.

Improving Git diffs for Lisp

nicolas@n16f.net (Nicolas Martyanoff) — Sun, 08 Jan 2023 18:00:00 +0000

All my code is stored in various Git repositories. When Git formats a diff between two objects, it generates a list of hunks, or groups of changes.

Each hunk can be displayed with a title which is automatically extracted. Git ships with support for multiple languages, but Lisp dialects are not part of it. Fortunately Git lets users configure their own extraction.

The first step is to identify the language using a pattern applied to the filename. Edit your Git attribute file at $HOME/.gitattributes and add entries for both Emacs Lisp and Common Lisp:

*.lisp diff=common-lisp
*.el diff=elisp

Then edit your Git configuration file at $HOME/.gitconfig and configure the path of the Git attribute file:

[core]
    attributesfile = ~/.gitattributes

Finally, set the regular expression used to match a top-level function name:

[diff "common-lisp"]
    xfuncname="^\\((def\\S+\\s+\\S+)"
    
[diff "elisp"]
    xfuncname="^\\((((def\\S+)|use-package)\\s+\\S+)"

For Lisp dialects, we do not just identify function names: it is convenient to identify hunks for all sorts of top-level definitions. We use a regular expression which captures the first symbol of the form and the name that follows.

Of course you can modifiy these expressions to identify more complex top-level forms. For example, for Emacs Lisp, I also want to identify use-package expressions.

You can see the result in all tools displaying Git diffs, for example in Magit with Common Lisp code:

Or for my Emacs configuration file:

Hunk titles, highlighted in blue, now contain the type and name of the top-level construction the changes are associated with.

A simple change, but one which really helps reading diffs.

Eshell key bindings and completion

nicolas@n16f.net (Nicolas Martyanoff) — Fri, 06 Jan 2023 18:00:00 +0000

I have been having completion issues in Eshell for some time, and I decided to fix it. Of course the yak shaving is always strong with Emacs, so it ended up being a bit more complicated than expected.

Defining keys with use-package

Eshell key bindings are stored in eshell-mode-map. For quite some time, Eshell has been initializing this value to nil, making it impossible to define keys before Eshell has been started. As initially reported, the workaround was to call define-key in a hook, eshell-mode-hook to be precise.

Apparently Emacs 28.1 fixed it. However to be able to use the key binding mechanism of use-package, eshell-mode-map must actually be defined. The solution is to load the right module before use-package configures key bindings, therefore using :init and not :config.

For example:

(use-package eshell
  :init
  (require 'esh-mode) ; eshell-mode-map

  :bind
  (("C-x s" . g-eshell)
   :map eshell-mode-map
   (("C-l" . 'g-eshell-clear)
    ("C-r" . helm-eshell-history))))

Configuring Company for completion

Eshell supports the standard completion-at-point completion system, the function being bound to <tab> by default. If you are using Helm, it will be automatically used to present the list of possible completions. This is a bit cumbersome: I want completion to be inline, not in a separate buffer in the bottom of the screen.

Since I use Company, I can instead bind <tab> to company-complete:

(define-key eshell-mode-map (kbd "<tab>") 'company-complete)

Fixing completion of commands in subdirectories

As it turns out, completion works as intended for global commands, but fails when calling a command in a subdirectory. For example, when completing ./utils/create-blog-post, completion-at-point will replace the line by create-blog-post, which is of course incorrect. This bug is also present when completing the absolute path of an executable.

After spending way too much time on Google and in the em-cmpl module, which handles completion for Eshell, it turned out to be a known bug, introduced almost two years ago.

The workaround is to redefine eshell--complete-commands-list as it was before the commit:

(defun eshell--complete-commands-list ()
  "Generate list of applicable, visible commands."
  (let ((filename (pcomplete-arg)) glob-name)
    (if (file-name-directory filename)
        (if eshell-force-execution
            (pcomplete-dirs-or-entries nil #'file-readable-p)
          (pcomplete-executables))
      (if (and (> (length filename) 0)
               (eq (aref filename 0) eshell-explicit-command-char))
          (setq filename (substring filename 1)
                pcomplete-stub filename
                glob-name t))
      (let* ((paths (eshell-get-path))
             (cwd (file-name-as-directory
                   (expand-file-name default-directory)))
             (path "") (comps-in-path ())
             (file "") (filepath "") (completions ()))
        ;; Go thru each path in the search path, finding completions.
        (while paths
          (setq path (file-name-as-directory
                      (expand-file-name (or (car paths) ".")))
                comps-in-path
                (and (file-accessible-directory-p path)
                     (file-name-all-completions filename path)))
          ;; Go thru each completion found, to see whether it should
          ;; be used.
          (while comps-in-path
            (setq file (car comps-in-path)
                  filepath (concat path file))
            (if (and (not (member file completions)) ;
                     (or (string-equal path cwd)
                         (not (file-directory-p filepath)))
                     (if eshell-force-execution
                         (file-readable-p filepath)
                       (file-executable-p filepath)))
                (setq completions (cons file completions)))
            (setq comps-in-path (cdr comps-in-path)))
          (setq paths (cdr paths)))
        ;; Add aliases which are currently visible, and Lisp functions.
        (pcomplete-uniquify-list
         (if glob-name
             completions
           (setq completions
                 (append (if (fboundp 'eshell-alias-completions)
                             (eshell-alias-completions filename))
                         (eshell-winnow-list
                          (mapcar
                           (lambda (name)
                             (substring name 7))
                           (all-completions (concat "eshell/" filename)
                                            obarray #'functionp))
                          nil '(eshell-find-alias-function))
                         completions))
           (append (and (or eshell-show-lisp-completions
                            (and eshell-show-lisp-alternatives
                                 (null completions)))
                        (all-completions filename obarray #'functionp))
                   completions)))))))

I have no idea what is wrong in the new implementation of the function, but hopefully someone can find a fix to be merged before Emacs 29 is released.

Org-mode headline tips

nicolas@n16f.net (Nicolas Martyanoff) — Mon, 02 Jan 2023 18:00:00 +0000

Org-mode is what got me into Emacs in the first place. I use it to take notes, keep track of what I have to do and plan projects.

An Org file is a tree of elements, each one starting with a headline. Manipulating headlines can be confusing; here are a few tips that you may find helpful.

Do not split lines when adding headlines

At any point, you can insert a new headline below the current one using the C-<return> shortcut, which calls org-insert-heading. However if the cursor is at the middle of a line (headline or content), Org will split it and add the end of the line to the new headline.

~~Quite annoying, really. Fortunately there is a function to add a headline without splitting the currentline, org-insert-heading-respect-content. I simply bind it to C-<return>:~~

~~Obviously I want the same behaviour for C-M-<return> which inserts a TODO headline:~~

Edit (2023-01-07)
As it turns out, a simpler way to do that is to set org-insert-heading-respect-content:

(setq org-insert-heading-respect-content t)

No need to rebind keys.

Jump quickly to the right headline

Some of my Org files are quite long. For example, my main development tracking file contains more than 700 lines.

I never really thought about navigation, and usually either search for elements by title with C-s or move between folded items manually (I use org-startup-folded so that all items are folded by default).

As it turns out, Org has a buitin navigation system triggered by C-c C-j. At first I could not see how it could help; but then I realized that this feature can use native Emacs completion. This is incredibly useful when combined with a vertical completion framework such as Helm, making it easy to jump to any headline just by typing a couple characters.

In order to do that, we enable completion with org-goto-interface, disable the multi-steps navigation system, and set the maximum depth level used to select headlines which will be listed during completion.

(setq org-goto-interface 'outline-path-completion)
(setq org-outline-path-complete-in-steps nil)
(setq org-goto-max-level 2)

Stop highlighting TODO headlines

By default, Org highlights the entire headline for TODO items, not just the TODO or DONE keyword.

I believe this is a mistake: headlines mark the hierarchy of items, and having multiple red or green headlines is confusing. This behaviour can be configured with two settings:

(setq org-fontify-todo-headline nil)
(setq org-fontify-done-headline nil)

This way, headlines keep their original color (defined by org-level-* faces), and keywords only are highlighted with the org-todo and org-done faces.

Configuring SLIME cross-referencing

nicolas@n16f.net (Nicolas Martyanoff) — Wed, 28 Dec 2022 18:00:00 +0000

The SLIME Emacs package for Common Lisp supports cross-referencing: one can list all references pointing to a symbol, move through this list and jump to the source code of each reference.

Removing automatic reference jumps

While cross-referencing is very useful, the default configuration is frustrating: moving through the list in the Emacs buffer triggers the jump to the reference under the cursor. If you are interested in a reference in the middle of the list, you will have to move to it, opening multiple buffers you do not care about as a side effect. I finally took the time to fix it.

Key bindings for slime-ref-mode mode are stored in the slime-xref-mode-map keymap. After a quick look in slime.el, it is easy to remove bindings for slime-xref-prev-line and slime-xref-next-line:

(define-key slime-xref-mode-map (kbd "n") nil)
(define-key slime-xref-mode-map [remap next-line] nil)
(define-key slime-xref-mode-map (kbd "p") nil)
(define-key slime-xref-mode-map [remap previous-line] nil)

If you are using use-package, it is even simpler:

(use-package slime
  (:map slime-xref-mode-map
      (("n")
       ([remap next-line])
       ("p")
       ([remap previous-line]))))

Changing the way references are used

SLIME supports two ways to jump to a reference:

With return or space, it spawns a buffer containing the source file and close the cross-referencing buffer.
With v, it spawns the source file buffer but keeps the cross-referencing buffer open and keeps it current.

This is not practical to me, so I made a change. The default action, triggered by return, now keeps the cross-referencing buffer open and switches to the source file in the same window. This way, I can switch back to the cross-referencing buffer with C-x b to select another reference without spawning buffers in other windows (I do not like having my windows hijacked by commands).

To do that, I need a new function:

(defun g-slime-show-xref ()
  "Display the source file of the cross-reference under the point
in the same window."
  (interactive)
  (let ((location (slime-xref-location-at-point)))
    (slime-goto-source-location location)
    (with-selected-window (display-buffer-same-window (current-buffer) nil)
      (goto-char (point))
      (g-recenter-window))))

Note the use of g-recenter-window, a custom function to move the current point at eye level. Feel free to use the builtin recenter function instead.

I then bind the function to return and remove other bindings:

(define-key slime-xref-mode-map (kbd "RET") 'g-slime-show-xref)
(define-key slime-xref-mode-map (kbd "SPC") nil)
(define-key slime-xref-mode-map (kbd "v") nil)

Much better now!

Replacing Projectile by Project

nicolas@n16f.net (Nicolas Martyanoff) — Tue, 27 Dec 2022 18:00:00 +0000

I have been using Projectile for years, and it served me well. While Projectile is really useful, it is a third party package made of more than 8000 lines of code. Since the built-in Project module has received lots of improvements since Emacs 28.1, I decided to give it a try and see if I could remove Projectile from my Emacs setup.

Key binding

The Project keymap is bound to C-x p by default, but I am used to C-c p. This is easy to change, for example with use-package:

(use-package project
  :bind-keymap
  (("C-c p" . project-prefix-map)))

Switching between projects

By default, the project switching command uses a prompt letting you select a command to run for the selected project. If you always use the same command, you can configure it, allowing to switch project without any prompt.

For example, to always display the project directory with Dired:

(setq project-switch-commands 'project-dired)

Search using Helm and AG

The Projectile feature I used the most is incremental project search using helm-projectile and helm-ag.

Adding this feature to Project is not that hard. Let us write the search function:

(defun g-project-search ()
  (interactive)
  (let ((project (project-current t)))
    (helm-do-ag (project-root project))))

And bind it to C-c p s (replacing project-shell, a command I do not use):

(use-package project
  :bind
  (:map project-prefix-map
        (("s" . 'g-project-search))))

One thing to be aware of: helm-ag does not take patterns returned by project-ignores into consideration when filtering results. It is not that much of a problem to me, since ag already supports .gitignore files

But it would be an improvement to use project-ignores and pass the pattern list to ag.

In the mean time, I can also remove the helm-projectile dependency.

Starting Eshell

While both Projectile and Project can run Eshell in the current project, I want the ability to start it the same way whether I am in a project or not. Therefore I have a small function looking for a current project and running Eshell the right way. Migrating to Project is easy:

(defun g-eshell ()
  "Start eshell at the root of the current project, or in the
current directory if the current buffer is not part of a
project."
  (interactive)
  (if (project-current)
      (project-eshell)
    (eshell)))

I bind it to C-x s (“s” being for “shell”):

(global-set-key (kbd "C-x s") 'g-eshell)

Conclusion

The result is quite satisfying. It may not do everything Projectile supports, but the features I use are all implemented. As a result, my Emacs setup is simpler and lighter, relying on one less external dependency.

Clearing the Eshell Buffer

nicolas@n16f.net (Nicolas Martyanoff) — Sat, 24 Dec 2022 18:00:00 +0000

I have used the C-l shortcut in ZSH to clear the screen for as long as I can remember. When I started using Eshell, one of the first problems I encountered was the inability to clear the buffer. I like to remove the output of previous commands to be able to focus on the current one, so let us script Emacs once again.

After a quick look to the eshell module, it seems we can do that in two steps:

Clear the entire buffer with (eshell/clear t).
Recreate the prompt with (eshell-emit-prompt).

A non-obvious edge case is that it should be possible to hit C-l while typing a command without losing what was typed. For that, we have to keep a copy of the input content before clearing the screen and reinsert it after.

The result is the following function:

(defun g-eshell-clear ()
  (interactive)
  (let ((input (eshell-get-old-input)))
    (eshell/clear t)
    (eshell-emit-prompt)
    (insert input)))

We then write a hook which binds the function to C-l:

(defun g-eshell-init-keys ()
  (define-key eshell-mode-map (kbd "C-l") 'g-eshell-clear))

And register the hook:

(add-hook 'eshell-mode-hook 'g-eshell-init-keys)

Note that clearing the screen means removing shell history from the buffer. Not really an issue: instead of looking in the buffer, I use helm-eshell-history which is much more convenient.

Investigating a FFAP issue in Emacs

nicolas@n16f.net (Nicolas Martyanoff) — Thu, 22 Dec 2022 18:00:00 +0000

I just encountered the strangest behaviour in Emacs. I was editing a shell script, hit C-x b to switch to another buffer, and Emacs suddenly froze with the following message:

Pinging <hostname> (Commercial)...

Where <hostname was the hostname of a server used in the script.

I really do not like the idea of my text editor pinging random servers just because their name appears in a buffer, so I went to the bottom of it.

Since I had no idea what caused this behaviour, I simply ran grep for the “Pinging” message in the Emacs source repository and found it in the ffap-machine-p function. To find out how it was called, I enabled tracing, making sure to print the current backtrace for each trace:

(trace-function 'ffap-machine-p nil 'backtrace)

This yielded the following (truncated) trace:

ffap-machine-p("pkg.exograd.com")
ffap-machine-at-point()
ffap-guesser()
ffap-guess-file-name-at-point()
run-hook-with-args-until-success(ffap-guess-file-name-at-point)
helm-guess-filename-at-point()
helm-fuzzy-highlight-matches(…)
helm--collect-matches(…)
helm-update(nil)
helm-read-from-minibuffer(nil nil nil nil nil nil nil)
helm-internal(…)
apply(helm-internal (…))
helm((helm-source-buffers-list helm-source-buffer-not-found) …)
apply(helm ((helm-source-buffers-list helm-source-buffer-not-found) …)
helm(…)
helm-buffers-list()
funcall-interactively(helm-buffers-list)
call-interactively(helm-buffers-list nil nil)
command-execute(helm-buffers-list)

Apparently, Helm tries to guess if the cursor is pointing to a filename using the ffap (for “find file at point”) module. For some reason, it does not just identify paths, but also URLs and machine names. And the default strategy to check if a machine is “real” and “reachable” (as documented in ffap-machine-p is to ping it.

This kind is insanity is really disappointing from Emacs: a text editor should absolutely not use the network in such a hidden way by default.

The behaviour of ffap-machine-p depends on three settings:

ffap-machine-p-local: what to do if the hostname does not have a domain.
ffap-machine-p-known: what to do if the hostname has a known domain.
ffap-machine-p-unknown: what to do if the hostname has an unknown domain.

For all these variables, the value is either accept, to indicate that the hostname is a valid hostname, reject to indicate it is not, or ping to check by sending a ping message.

The way to fix this mess is to accept local and known hostnames while rejecting unknown ones:

(setq ffap-machine-p-local 'accept)
(setq ffap-machine-p-known 'accept)
(setq ffap-machine-p-unknown 'reject)

If you are curious, ffap uses mail-extr-all-top-level-domains in the mail-extr module to assess if a domain is “known”. This function will identify most common TLDs, but has never been updated for modern TLDs. Something to keep in mind if you want to use it.

Fixing unquote-splicing behaviour with Paredit

nicolas@n16f.net (Nicolas Martyanoff) — Tue, 13 Dec 2022 18:00:00 +0000

Paredit is an Emacs package for structural editing. It is particularly useful in Lisp languages to manipulate expressions instead of just characters.

One of the numerous little features of Paredit is the automatic insertion of a space character before a delimiting pair. For example, if you are typing (length, typing ( will have Paredit automatically insert a space character before the opening parenthesis, to produce the expected (length ( content.

Paredit is smart enough to avoid doing so after quote, backquote or comma characters, but not after an unquote-splicing sequence (,@) which is quite annoying in languages such as Scheme or Common Lisp. As almost always in Emacs, this behaviour can be customized.

Paredit decides whether to add a space or not using the paredit-space-for-delimiter-p function, ending up with applying a list of predicates from paredit-space-for-delimiter-predicates.

Let us add our own. For more flexibility, we will start by defining a list of prefixes which are not to be followed by a space:

(defvar g-paredit-no-space-prefixes (list ",@"))

We then write our predicate which simply checks if we are right after one of these prefixes:

(defun g-paredit-space-for-delimiter (endp delimiter)
  (let ((point (point)))
    (or endp
        (seq-every-p
         (lambda (prefix)
           (and (> point (length prefix))
                (let ((start (- point (length prefix)))
                      (end point))
                  (not (string= (buffer-substring start end) prefix)))))
         g-paredit-no-space-prefixes))))

Finally we add a Paredit hook to append our predicate to the list:

(defun g-init-paredit-space-for-delimiter ()
  (add-to-list 'paredit-space-for-delimiter-predicates
               'g-paredit-space-for-delimiter))

(add-hook 'paredit-mode-hook 'g-init-paredit-space-for-delimiter)

Not only does it fix the problem for unquote-slicing, but it makes it easy to add new prefixes. For example I immediately added #p (used for pathnames in Common Lisp, e.g. #p"/usr/bin/") to the list.

SLIME compilation tips

nicolas@n16f.net (Nicolas Martyanoff) — Mon, 12 Dec 2022 18:00:00 +0000

I recently went back to Common Lisp to solve the daily problems of the Advent of Code. Of course it started with installing and configuring SLIME, the main major mode used for Common Lisp development in Emacs.

The most useful feature of SLIME is the ability to load sections of code into the Common Lisp implementation currently running. One can use C-c C-c to evaluate the current top-level form, and C-c C-k to reload the entire file, making incremental development incredibly convenient.

However I found the default configuration frustrating. Here are a few tips which made my life easier.

Removing the compilation error prompt

If the Common Lisp implementation fails to compile the file, SLIME will ask the user if they want to load the fasl file (i.e. the compiled form of the file) anyway.

I cannot find a reason why one would want to load the ouput of a file that failed to compile, and having to decline every time is quite annoying.

Disable the prompt by setting slime-load-failed-fasl to 'never:

(setq slime-load-failed-fasl 'never)

Removing the SLIME compilation buffer on success

When compilation fails, SLIME creates a new window containing the diagnostic reported by the Common Lisp implementation. I use display-buffer-alist to make sure the window is displayed on the right side of my three-column split, and fix my code in the middle column.

However if the next compilation succeeds, SLIME updates the buffer to indicate the absence of error, but keeps the window open even though it is not useful anymore, meaning that I have to switch to it and close it with q.

One can look at the slime-compilation-finished function to see that SLIME calls the function referenced by the slime-compilation-finished-hook variable right after the creation or update of the compilation buffer. The default value is slime-maybe-show-compilation-log which does not open a new window if there is no error, but does not close an existing one.

Let us write our own function and use it:

(defun g-slime-maybe-show-compilation-log (notes)
  (with-struct (slime-compilation-result. notes successp)
      slime-last-compilation-result
    (when successp
      (let ((name (slime-buffer-name :compilation)))
        (when (get-buffer name)
          (kill-buffer name))))
    (slime-maybe-show-compilation-log notes)))
    
(setq slime-compilation-finished-hook 'g-slime-maybe-show-compilation-log)`

Nothing crazy here, we obtain the compilation status (in a very SLIME-specific way, with-struct is not a standard Emacs Lisp macro) and kill the compilation buffer if there is one while compilation succeeded.

Making compilation less verbose

Common Lisp specifies two variables, *compile-verbose* and *load-verbose*, which control how much information is displayed during compilation and loading respectively.

My implementation of choice, SBCL, is quite chatty by default. So I always set both variables to nil in my $HOME/.sbclrc file.

However SLIME forces *compile-verbose*; this is done in SWANK, the Common Lisp part of SLIME. When compiling a file, SLIME instructs the running Common Lisp implementation to execute swank:compile-file-for-emacs which forces *compile-verbose* to t around the call of a list of functions susceptible to handle the file. The one we are interested about is swank::swank-compile-file*.

First, let us write some Common Lisp code to replace the function with a wrapper which sets *compile-verbose* to nil.

(let ((old-function #'swank::swank-compile-file*))
  (setf (fdefinition 'swank::swank-compile-file*)
        (lambda (pathname load-p &rest options &key policy &allow-other-keys)
          (declare (ignore policy))
          (let ((*compile-verbose* nil))
            (apply old-function pathname load-p options)))))

We save it to a file in the Emacs directory.

In Emacs, we use the slime-connected-hook hook to load the code into the Common Lisp implementation as soon as Slime is connected to it:

(defun g-slime-patch-swank-compilation-function ()
  (let* ((path (expand-file-name "swank-patch-compilation-function.lisp"
                                 user-emacs-directory))
         (lisp-path (slime-to-lisp-filename path)))
    (slime-eval-async `(swank:load-file ,lisp-path))))
    
(add-hook 'slime-connected-hook 'g-slime-patch-swank-compilation-function)

Quite a hack, but it works.

Eye level window centering in Emacs

nicolas@n16f.net (Nicolas Martyanoff) — Sun, 30 Oct 2022 18:00:00 +0000

In the default Emacs configuration, C-l is bound to recenter-top-bottom. This function has an interesting behaviour: when called once, it scrolls the current window so that the current line appears at the middle of the screen. When called several times in a row, it cycles through three scrolling position: middle, top and bottom.

This has always bugged me because I only need one of the three behaviours, and none of these positions are useful to me. In pratice, I want to focus on the line I am working on, which means putting it at eye level. In my case at roughly 20% of the top of the window.

Fortunately Emacs is easy to customize. As it turns out, recenter-top-bottom is based on recenter which is easy to use:

(defcustom g-recenter-window-eye-level 0.2
  "The relative position of the line considered as eye level in the
current window, as a ratio between 0 and 1.")

(defun g-recenter-window ()
  "Scroll the window so that the current line is at eye level."
  (interactive)
  (let ((line (round (* (window-height) g-recenter-window-eye-level))))
    (recenter line)))

(global-set-key (kbd "C-l") 'g-recenter-window)

Exporting 1Password data for backup

nicolas@n16f.net (Nicolas Martyanoff) — Sun, 26 Jun 2022 18:00:00 +0000

Storing credentials and confidential data in a secure location is important, and I have been happy with 1Password for years.

But having all this critical information stored in a single location is dangerous; losing access to my 1Password account would be quite annoying, forcing me to reinitialize passwords on dozens of websites and making me lose a lot of information.

Fortunately it is possible to export all data from 1Password. It does not seem to be doable directly from their web interface, but their command-line tool supports it.

You will need first to connect your account. For example for a personal account:

op account add --address my.1password.com --email YOUR_EMAIL_ADDRESS

Note that you will not stay signed-in, and will have to re-enter your password later using: eval $(op signin).

There is no command to directly export all entries, but it can be done with a small script. If you are not familiar with GPG, feel free to refer to my GnuPG introduction.

#!/bin/sh

set -eu
set -o pipefail

# Configuration
gpg_key_id="YOUR_GPG_KEY_ID"

# Command line
if [ $# -lt 1 ]; then
    echo "usage: $0 <output-file>" >&2
    exit 1
fi

output_file=$1

# Keep permissions tight
umask 177

# Sign in to the 1Password account
eval $(op signin)

# Create a temporary file to store the list of item ids (this list does not
# contain any confidential data).
item_file=$(mktemp)
trap "rm -f $item_file" EXIT

# Export a list containing the identifier and vault identifier of each item
op --format json item list | jq -r '.[] | .id + " " + .vault.id' >$item_file

# Export all items, encrypt all data and store them in the output file
while read item_id vault_id; do
    op --format json item get $item_id --vault $vault_id
done < $item_file | gpg --encrypt --sign --recipient $gpg_key_id >|$output

The resulting file can then be stored anywhere, without any specific protection since it is encrypted.

Using GnuPG

nicolas@n16f.net (Nicolas Martyanoff) — Sat, 29 Jan 2022 18:00:00 +0000

Edit (2023-02-07)
Updated to mention subkey selection during renewal.

Introduction

I recently tried to learn more about GnuPG and it turned out to be more difficult than expected. So here are my notes regarding various GnuPG operations.

GnuPG 2.2.32 was used for all commands.

Configuration

GnuPG

The gpg command line tool looks for $HOME/.gnupg/gpg.conf.

Options are provided using the command line arguments used by the gpg program without the -- prefix. For example:

verbose

causes GnuPG to behave as if the --verbose option was passed to every command.

I went with the following configuration:

# Do not print any greeting message
no-greeting

# Do not include comment or version numbers in signatures
no-comments
no-emit-version

# Display long key ids to minimize collision risks
keyid-format long

# Set preferences when signing and encrypting
# Use gpg --version to see available algorithms
personal-digest-preferences SHA256
personal-cipher-preferences AES256
personal-compress-preferences ZLIB

# Use AES256 as default cipher
cipher-algo AES256

# Use SHA256 as default digest
digest-algo SHA256
cert-digest-algo SHA256

# Use ZLIB as default compression algorithm
compress-algo ZLIB

# Always export in ASCII format by default
armor

Password prompts

GnuPG uses the pinentry collection of programs to prompt for passwords. We can configure the GPG agent to always use the terminal version of pinentry instead of spawning a graphical prompt.

The agent is configured in $HOME/.gnupg/gpg-agent.conf; to use the terminal prompt, set the following option:

pinentry-program /usr/bin/pinentry-tty

Note that the agent has to be restarted after any modification:

gpgconf --reload gpg-agent

Managing keys

Creating a key

Use the --full-gen-key option to create a new key. The --gen-key option can also be used but offer less control on the process.

The --expert option can also be used to access more advanced features such as ECC support.

The process is interactive.

Once created, the new key should be visible in the list of keys which can be visualized with gpg --list-keys.

Using multiple identities

A key can contain multiple identities. Edit the key with gpg --edit-key <key-id> and use the adduid command to add a new identity. You then need to trust the new identity. Use the uid <n> command to select the new uid, and run the trust command to select the trust level (probably ultimate since this is your own key). Make sure to use the primary command to select the current identity as the main one. Use the save command to write the modification and exit.

Note that modifying the list of identities changes the public key. If you published it, you will need to update it.

Deleting a key

Keys can be deleted using gpg --delete-secret-keys <key-id> and gpg --delete-keys <key-id>. Note that the secret key must be deleted before the associated public key.

Using subkeys

Newly created keys contain a public key, a secret key which can be used for signing and certifying other signatures, and a secret subkey which can be used for encryption.

It is a good idea to store the signing secret key in a safe location and to generate new signing subkeys for each device which will need to sign content. This way, if a device is compromised, one can revoke the associated signing subkey and generate a new one without compromising the primary identity which is guaranteed by the primary signing key.

To create a new subkey, edit the key with gpg --edit-key <keyid> and use the addkey command. Again, the --expert option can be passed to the --edit-key command to be able to create ECC keys. The addkey command will ask for a kind of key; to create a signing sub key, opt to “set your own capabilities”, and select the sign and authenticate capabilities. Use the save command to write the modification and exit.

At this point, it is possible to export both the new signing subkey and the encryption subkey and import then on the new device. Take care too use the ! suffix to only export these subkeys, as described in the next section. For example:

gpg --export-secret-keys <signing-subkey-id>! <encryption-subkey-id>!

If you want to remove the primary secret key from the local GnuPG directory, first obtain its keygrip using gpg --with-keygrip --list-secret-keys, then delete the key stored in $HOME/.gnupg/private-keys-v1.d/<keygrip>.key. You can check that the primary secret key is not here anymore: when listing secret keys, the deleted key type will have the # suffix, e.g. sec#.

Exporting keys

The most obvious way to share public keys is to export them from the local keyring. The keyring is stored in $HOME/.gnupg/pubring.kbx.

The gpg --export <user-id> command outputs one or more public keys associated with a user id. Note that there are multiple way to select a user id. See the official documentation for more information.

Using the key id or the email address is the simplest way to select the public key.

Secret key exports works the same way but use the --export-secret-key command. Note that exporting a subkey will by default export the entire secret key. Add a ! suffix to the key id to export a single subkey.

When backing up keys, use --export-options backup to include all necessary data for later restoration.

Publishing a key

Public keys can be published on internet servers to make it easier for other people to download them.

Note that the presence of a key on a public server does not imply it actually belong to the person identified by the key: anyone can generate keys using any name or email address.

Also remember that published keys can never be deleted.

To publish a key, simply use gpg --keyserver <address> --send-keys <key-id>.

Importing a key

Public and private keys can be imported in the local keyring, making them available for verification or decryption. Simply gpg --import <file>. It is also possible to pipe the public key directly into gpg --import.

Revoking a key

Compromised keys must be revoked to signal that someone other than the owner of the key has access to it.

Revocation uses a revocation certificate which is generated from the private key using gpg --gen-revoke <key-id>. The command is interactive and asks for the reason of the revocation.

The revocation certificate can then be imported or published like any other public key. After importing a revocation certificate in a keyring, the key will be marked as revoked.

Renewing a key

If a key has an expiration date, it may need to be renewed. This can be done by editing the key. Start the edition shell using gpg --edit-key <key-id>, and select the subkey you want to modify using the key <index> command. Then run the expire interactive command. You can then write the modification and exit with save.

Do not forget to backup the new version of the private key and to publish the new version of the public key.

Note that you should generate and backup a new revocation certificate.

Using keys

Encrypting a file

Data are encrypted using gpg --encrypt --recipient <key-id>, and decrypted using gpg --decrypt. Encrypted messages contain the key id; decrypting the file requires the presence of the associated secret key in the local GnuPG directory.

Signing a file

Data are signed using gpg --sign <file>. The output contains both the data and the signature.

Signed data are verified using gpg --verify.

It is also possible to generate a separate — or detached — signature using gpg --detach-sig, e.g.:

gpg --output my-file.asc --detach-sig my-file
gpg --verify my-file.asc my-file

Default configuration for the Erlang logger

nicolas@n16f.net (Nicolas Martyanoff) — Sun, 15 Nov 2020 18:00:00 +0000

The Erlang logger application introduced in OTP 21 is extremely flexible, but the default settings are not the most practical for development purposes:

The default log level is notice, meaning that info and debug messages will not be printed.
Log messages with a domain other than [otp] will not be printed.
Each printed log message will start with a cumbersome banner, e.g. =NOTICE REPORT==== 15-Nov-2020::11:20:10.421265 ===.

While it is possible to change logger settings in the system configuration file, it means that it has to be done for each project you work on.

Fortunately, Erlang has the ability to load and execute code from a file at $HOME/.erlang. For example, I use the following code to configure the logger:

LogHandler = #{level => debug,
               filter_default => log,
               filters => [{remote_group_leader,
                            {fun logger_filters:remote_gl/2, stop}},
                           {progress,
                            {fun logger_filters:progress/2, stop}},
                           {sasl,
                            {fun logger_filters:domain/2,
                             {stop, sub, [otp, sasl]}}}],
               formatter => {logger_formatter,
                             #{single_line => false,
                               template =>
                                   [domain, " ", level, ": ", msg, "\n"]}}},
application:set_env(kernel, logger,
                    [{handler, default, logger_std_h, LogHandler}]),
application:set_env(kernel, logger_level, debug),
logger:set_handler_config(default, LogHandler).

For development, I went with the following settings:

All messages are logged. Debug logging can still be disabled on a per-project basis (using a local system configuration file) for applications which are too verbose.
Remote messages and progress messages are dropped, they are usually useless during development.
SASL messages are also dropped by default, they tend to be quite redundant.
We keep multi-line display, which is really useful to read stack traces.
Log messages are prefixed by the domain (see the documentation for more information about domains) and the log level.

Note that while this setup will work with erl, the Rebar3 shell requires some adjustments.

Rebar3 must be installed from source with a start script, because the packaged escript-based version does not load $HOME/.erlang. There is an open issue, but in the mean time, you can install Rebar3 from source:

Clone the repository.
Build Rebar3: ./bootstrap.
Install it locally: ./rebar3 local install. Note that ./rebar3 local upgrade will not use the code you just built, but will always fetch a stable release from the internet.
Add $HOME/.cache/rebar3/bin to your PATH environment variable.

There is also an issue with logger configuration reloading in the shell. I created a PR, but in the mean time you will need to apply this patch.

Finally, if you load extra system configuration files with the --config command line option, you will have to make sure that any logger modification performed in $HOME/.erlang is also applied to system configuration (e.g. with application:set_env/3), since Rebar3 will reload the logger configuration from the environment, ignoring any live setting.

Unknown type specifications with Dialyzer

nicolas@n16f.net (Nicolas Martyanoff) — Mon, 07 Sep 2020 18:00:00 +0000

I use Dialyzer a lot, and it has always been useful to expose various errors in my Erlang code.

Unfortunately, I recently discovered an unexpected behaviour: by default, Dialyzer ignores unknown types. It is very surprising and suddently makes me doubt any piece of Erlang code I ever wrote.

As it turns out, there is a Dialyzer flag to detect and report these unknown types. With Rebar3, it can be added to the dialyzer setting:

{dialyzer, [{warnings, [unknown]}]}.

Activating unknown type detection could lead to interesting Dialyzer failures, when a type is part of an unknown module. Care should be taken to list application dependencies in the application resource file.

In some cases, it is necessary to reference a type from a module whose application is not a dependency of the current project. In that case, the application in question can be added to the PLT with the plt_extra_apps option.

For example:

{dialyzer, [{plt_extra_apps, [public_key]},
            {warnings, [unknown]}]}.

Activating unknown type detection for all my Erlang projects has revealed various issues, so it is definitely an option I will keep enabled in the future.

Local CLHS access in Emacs

nicolas@n16f.net (Nicolas Martyanoff) — Wed, 01 Jan 2020 18:00:00 +0000

The CLHS, or Common Lisp HyperSpec, is one of the most important resource for any Common Lisp developer. It is derived from the official Common Lisp standard, and contains documentation for every aspect of the language.

While it is currently made available online by LispWorks, it can be useful to be able to access it locally, for example when you do not have any internet connection available.

For this purpose, LispWorks provides an archive which can be downloaded and browsed offline.

While the HyperSpec is valuable on its own, the SLIME Emacs mode provides various functions to make it even more useful.

I have found the following functions particularily useful:

slime-documentation-lookup, or C-c C-d h to browse the documentation associated with a symbol.
common-lisp-hyperspec-format, or C-c C-d ~, to lookup format control characters.
common-lisp-hyperspec-glossary-term, or C-c C-d g, to access terms in the glossary.

With the default configuration, Emacs will use the online HyperSpec. You can have it use a local copy by setting common-lisp-hyperspec-root to a file URI. For example, if you downloaded the content of the CLHS archive to ~/common-lisp/:

(setq common-lisp-hyperspec-root
  (concat "file://" (expand-file-name "~/common-lisp/HyperSpec/")))

And if you configure Emacs to use the EWW web browser, you can work with the CLHS without leaving your editor.

ASDF in Common Lisp scripts

nicolas@n16f.net (Nicolas Martyanoff) — Mon, 30 Dec 2019 18:00:00 +0000

The usual way to develop programs in Common Lisp is to use SLIME in Emacs, which starts an implementation and provides a REPL. When a program needs to be running in production, one can either execute it from source or compile it to an executable core, for example with sb-ext:save-lisp-and-die in SBCL.

While executable cores works well for conventional applications, they are less suitable for small scripts which should be easy to run without having to build anything.

Writing a basic script with SBCL is easy:

#!/bin/sh
#|
exec sbcl --script "$0" "$@"
|#

(format t "Hello world!~%")

Since UNIX shebangs cannot be used to run commands with more than one argument, it is impossible to call SBCL directly (it requires the --script argument, and #!/usr/bin/env sbcl --script contains two arguments). However it is possible to start as a simple shell script and just execute SBCL with the right arguments. And since we can include any shell commands, it is possible to support multiple Common Lisp implementations depending on the environment.

This method works. But if your script has any dependency, configuring ASDF can be tricky. ASDF can pick up system directory paths from multiple places, and you do not want your program to depend on your development environment. If you run your script in a CI environment or a production system, you will not have access to your ASDF configuration and your systems.

Fortunately, ASDF makes it possible to manually configure the source registry at runtime using asdf:initialize-source-registry, giving you total control on the places which will be used to find systems.

For example, if your Common Lisp systems happen to be stored in a systems directory at the same level as your script, you can use the :here directive:

#!/bin/sh
#|
exec sbcl --script "$0" "$@"
|#

(require 'asdf)

(asdf:initialize-source-registry
 `(:source-registry :ignore-inherited-configuration
                    (:tree (:here "systems"))))

And if you store all your systems in a Git repository, you can use submodules to include a systems directory in every project, making it simple to manage the systems you need and their version. Additionally, anyone with an implementation installed, SBCL in this example, can now execute these scripts without having to install or configure anything. This is quite useful when you work with people who do not know Common Lisp.

Of course, you can use the same method when building executables: just create a script whose only job is to setup ASDF, load your program, and dump an executable core. This way, you can make sure you control exactly which set of systems is used. And it can easily be run in a CI environment.

Persistent history for the Erlang shell

nicolas@n16f.net (Nicolas Martyanoff) — Mon, 25 Nov 2019 18:00:00 +0000

While the erl application is really useful to tinker with Erlang modules, the lack of a persistent history has always been annoying. I recently discovered via Adopting Erlang that the history can be made persistent since Erlang 20 with specific flags, which can be added to the ERL_AFLAGS environment variable to have them enabled by default.

Shell history is handled by the group_history module; since this module is part of the kernel application, the configuration flags are provided with the -kernel command line argument (see the [erl documentation](application:get_env(kernel, shell_history_drop)), specifically the -Application Par Val syntax. An interesting property of this mechanism is that parameters are passed as Erlang terms. Therefore a string should be passed in shell as '"foo bar"'.

The module will simply use application:get_env(kernel, Name) to read these parameters.

The kernel_app documentation presents the following parameters:

shell_history: whether to enable or disable permanent history.
shell_history_drop: a list of commands to be omitted from the history.
shell_history_file_bytes: the maximum size of the history.
shell_history_path: the path of the directory to write history to.

It is really simple to add these flags to ERL_AFLAGS, for example in $HOME/.zshenv for the zsh shell:

ERL_AFLAGS="+pc unicode"
ERL_AFLAGS="$ERL_AFLAGS -kernel shell_history enabled"
ERL_AFLAGS="$ERL_AFLAGS -kernel shell_history_path '\"$HOME/.erl_history\"'"

export ERL_AFLAGS

I only found two small issues with this feature:

There is a small delay between the time an entry is saved and the moment is is actually written to disk. Therefore quitting quickly after entering a command may result in the entry missing from the history.
History searching with C-r seems to work, but is far less powerful than the readline feature. Good enough I guess.

Reading RFC documents in Emacs

nicolas@n16f.net (Nicolas Martyanoff) — Sun, 02 Jun 2019 18:00:00 +0000

I am used to read RFC documents in Emacs by opening them as simple text files. But I was always annoyed by the impossibility to follow references to other documents. Additionally, searching for the right RFC usually involve going back to the web browser and Google while I would prefer staying in Emacs.

Since I enjoy Helm so much, I thought it would be easy to parse the index of all RFC documents, and create a browser allowing me to pick a RFC. And since I was writing some elisp, I ended up writing a major mode to highlight parts of RFC documents, including adding links when another RFC is mentioned.

So here is rfc-mode. It is quite simple for the time being, and I have a few ideas for more features, but it is already quite useful.

It is also available on MELPA.

Printing non-ascii characters in Erlang releases

nicolas@n16f.net (Nicolas Martyanoff) — Fri, 04 Jan 2019 18:00:00 +0000

I have been learning a lot about Erlang recently, and I stumbled upon a strange behaviour: printing strings containing non-ascii characters, e.g. "μs", using io:format and the ~ts control sequence behaves differently depending on the way the program is run:

In rebar3 shell, rebar3 run, or a release running with a console: "μs".
With a release running in foreground: "\x{3BC}s".

I first checked the shell environment, but os:getenv("LANG") correctly returns en_US.UTF-8 in both cases. I then suspected the Erlang VM was initialized with a different +pc flag in console mode, but adding +pc unicode to vm.args did not change anything.

Finally I ended up on the very detailed IO protocol documentation. Printing the IO configuration of the standard output, obtained with io:getopts(), yields different results depending on the environment: in interactive environments and with rebar3 run, the encoding parameter is set to unicode, but is set to latin1 in releases running in foreground.

As it turns out, configuring the standard IO device to output UTF-8 encoded strings is a simple as calling io:setopts([{encoding, unicode}]) when the application starts. This is definitely something I will add to all my applications from now on.

Buffer ordering with Helm

nicolas@n16f.net (Nicolas Martyanoff) — Sun, 21 Oct 2018 18:00:00 +0000

I have been using Emacs for some time, and ido has always been my favourite way to switch buffers.

But recently I discovered and started using Helm, a package based on incremental completion which can provide lots of interesting features. One of them being buffer switching.

The simplest way to use Helm to switch buffers is helm-buffers-list. Personally I use C-x b:

(global-set-key (kbd "C-x b") 'helm-buffers-list)

I really like that helm-buffer-list orders buffers using the last time they were used. However as soon as I start typing something to narrow the selection, Helm switches to a different ordering based on the length of buffer names. A bug report was submitted, but apparently, this is the intended behaviour.

Fortunately, it is possible to override this behaviour, as suggested in another Github issue. Ordering is done in the helm-buffers-sort-transformer function; we could replace it, but it is cleaner to advise it.

(defun nm-around-helm-buffers-sort-transformer (candidates source)
  candidates)

(advice-add 'helm-buffers-sort-transformer
            :override #'nm-around-helm-buffers-sort-transformer)

This way we keep the LRU-style ordering even when narrowing the selection.

Nicolas Martyanoff – Brain dump

Working with Common Lisp pathnames

Filenames and file paths

Pathnames

Special characters

Invalid characters

Pathname component case

Unspecific components

Namestrings

Native namestrings

Wildcards

Logical pathnames

Logical namestrings

Translation

Caveats

Recipes

Resolving a path

Transforming paths

Joining two paths

Finding files

Resolving tildes in paths

Controlling link opening in Emacs

Key bindings

Selecting a web browser

Links in Gnus messages

Links in the Circe IRC client

Taking control of your Go module paths

How packages are downloaded

Using GitHub Pages

Generating repository pages

Configuring the domain

Publishing pages

Updating modules

Result

Pagination for database objects

Limit and offset

SQL Cursors

Key-based pagination

Sorting

Interactive Common Lisp development

Variables

Functions

Symbols and packages

Classes

Generics and methods

Meta Object Protocol

Structures

Conditions

Types

Going further

Building Erlang applications the hard way

A minimal Erlang program

Writing the modules

Writing the application definition

Compiling and running the application

Building a release

Creating the release resource file

Generating the boot script

Generating a release package

Running the application

Conclusion

Reduce vs fold in Common Lisp

Introduction

Understanding REDUCE

Differences with fold

Implementing FOLDL

Replacing Ngrok with Frp

Configuring the server

Frp server

NGINX

Configuring the client

Done!

Making IELM More Comfortable

Adding Eldoc hints

Using Paredit

Making the command history persistent

Useful key bindings

Golang workspaces, at last!

Go modules are not ideal

Using workspaces

Understanding `REDUCE`

Implementing `FOLDL`