Nicolas Martyanoff – Brain dump About

Building Erlang applications the hard way

erlang

Edit (2023-06-19)
Updated to mention the +Bd option.

I was recently in a conversation involving Erlang, and it made me feel like revisiting the subject. Last time I used Erlang for a commercial application, I remember a lot of frustration with the tooling, due in part to the small size of the ecosystem. Today I decided to read the documentation and see how to build an Erlang program from scratch. Forget about Rebar3 and Erlang.mk, we are going to do this the hard way.

A minimal Erlang program

Let us start with a small program. We could of course just write a traditional Hello World printing a message and exiting, but we are using Erlang and Erlang is all about servers. So our program is a server printing a message to the standard output every second. We will build it according to OTP principles.

Writing the modules

We will need three modules.

First the application, whose only role is to start the supervisor:

-module(hello_app).

-behaviour(application).

-export([start/2, stop/1]).

start(_StartType, _Args) ->
  hello_sup:start_link().

stop(_State) ->
  ok.

Then the supervisor, with a single child:

-module(hello_sup).

-behaviour(supervisor).

-export([start_link/0]).
-export([init/1]).

start_link() ->
  supervisor:start_link(?MODULE, []).

init(_Args) ->
  Children = [#{id => hello_server,
                start => {hello_server, start_link, []}}],
  {ok, {#{}, Children}}.

And finally the server, printing a message every second:

-module(hello_server).

-behaviour(gen_server).

-export([start_link/0]).
-export([init/1, terminate/2, handle_call/3, handle_cast/2, handle_info/2]).

start_link() ->
  gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).

init([]) ->
  schedule_hello(),
  {ok, undefined}.

terminate(_Reason, _State) ->
  ok.

handle_call(_Msg, _From, State) ->
  {reply, unhandled, State}.

handle_cast(_Msg, State) ->
  {noreply, State}.

handle_info({hello, Message}, State) ->
  io:format("~ts~n", [Message]),
  schedule_hello(),
  {noreply, State}.

schedule_hello() ->
  erlang:send_after(1000, self(), {hello, <<"Hello world!">>}).

We store module files in a src directory.

Writing the application definition

In OTP, an application is a component made of a set of modules working together. Applications are defined with an application file; they contain an Erlang expression describing the application and end with the .app extension. OTP, more precisely the application controller, will use it to know how to start, stop and in general handle the application.

Since the application file is read at runtime, we will store it in a second directory named ebin, which will also be used to store BEAM code files.

Add the following content to ebin/hello.app:

{application, hello,
 [{description, "A hello world application."},
  {vsn, "1.0.0"},
  {modules,
   [hello_app,
    hello_sup,
    hello_server]},
  {registered,
   [hello_sup,
    hello_server]},
  {applications,
   [kernel,
    stdlib]},
  {mod, {hello_app, []}},
  {env, []}]}.

Again nothing complicated. Feel free to refer to the documentation for more information.

Compiling and running the application

At the lowest level, we need to be able to compile modules, transforming Erlang code into BEAM bytecode. We can do that just by calling erlc. Of course we probably do not want to call it manually for each module after each modification, so let us write a simple Makefile:

ERLC_OPTIONS ?= -Wall -Werror

ERL_SRC = $(wildcard src/*.erl)
ERL_OBJ = $(patsubst src/%.erl,ebin/%.beam,$(ERL_SRC))

all: build

build: $(ERL_OBJ)

ebin/%.beam: src/%.erl
	erlc $(ERLC_OPTIONS) -o $(dir $@) $<

clean:
	$(RM) $(wildcard ebin/*.beam)

.PHONY: all build clean

Careful about the -o option: while most compilers use it to provide the name of the destination file, erlc expects a directory but will not signal an error if it is a file, instead ignoring the option altogether.

Run make to compile your modules. We can now run our application. Start erl, using the -pa ebin option to add the ebin directory to the list of paths Erlang looks for code files, and start the hello application in the shell:

application:start(hello).

Barring any unexpected error, you should see the Hello world! message printed every second.

Of course you can also run it in a non-interactive way:

erl -noshell -pa ebin -eval "application:start(hello)"

Note that even with -noshell, there does not seem to be any way to tell the Erlang runtime to exit on SIGINT instead of opening a interactive “break handler” requiring additional input. Disappointing. You will notice that using C-c in a terminal, i.e. sending SIGINT to the Erlang runtime, starts the interactive break handler instead of stopping the program. You can use the -Bd option to exit on SIGINT like other programs. Thanks to Asabil for telling me about the -B option on Reddit!

At this point it is tempting to call it a day. After all we can define an application, build its modules and run it. But this approach is limited: we still need to install Erlang on the target server, ideally with the same version as the one used for development and tests. We could also be tempted to use code hot-swapping to upgrade the application without stopping it.

Erlang supports releases, a way of packaging multiple applications and the Erlang runtime itself into a self-contained archive. So let us build a release!

Building a release

Creating the release resource file

A release starts with a release resource file, an Erlang file describing the release and the application it will contain.

Our release is simple:

{release, {"hello", "1.0.0"},
 {erts, "13.2"},
 [{kernel, "8.5.4"},
  {stdlib, "4.3"},
  {hello, "1.0.0"}]}.

We define a released named hello with version 1.0.0, based on ERTS 13.2, and we list three applications: kernel, stdlib, and hello. Remember that kernel and stdlib must always be included.

You will notice that we need to provide the version for each component. This is really frustrating because they depend on the version of Erlang installed on your system, and you would expect systool to handle it automatically, but this is how it is. Clearly the release resource file is something to be generated automatically. In the mean time, you will need to adapt the version numbers of the example (those are for Erlang 25.3.2).

Generating the boot script

While the release resource file defines what is in the release, the boot script explains how to start it. Again, it contains Erlang expressions, and it can quickly get complicated (see the documentation for more information). Furthermore, the Erlang release handler will not use the boot script directly, but instead a compiled version with the .boot file extension. Fortunately we do not have to write the script ourselves: systools will handle both generation and compilation.

To use systools, let us write a small Erlang program named generate-boot:

#!/usr/bin/env escript

main([ReleaseName]) ->
  Options = [no_warn_sasl, {path, ["ebin"]}, silent],
  case systools:make_script(ReleaseName, Options) of
    {ok, _Module, []} ->
      ok;
    {ok, Module, Warnings} ->
      log_error("warning: ~ts", [Module:format_warning(Warnings)]);
    {error, Module, Errors} ->
      log_error("error: ~ts", [Module:format_error(Errors)]),
      halt(1)
  end;
main(_Args) ->
  log_error("Usage: ~ts <release-name>~n", [escript:script_name()]),
  halt(1).

log_error(Format, Args) ->
  io:format(standard_error, Format, Args).

We expect the script to be called with one argument being the name of the release, and all we do is to call systools:make_script to generate the script file and compile it to a boot file.

We provide three options:

  • no_warn_sasl to tell systools not to print any warning because we did not include the SASL application (unnecessary since we do not deal with release upgrading in this article).
  • {path, ["ebin"]} to look in the ebin directory for application definition files.
  • silent to return warnings and errors instead of printing them directly. This is necessary because we want to log to the standard error output and exit with status code 1.

We then update the Makefile to be able to build the boot file:

RELEASE = hello

$(RELEASE).boot: $(RELEASE).rel ebin/$(RELEASE).app
	./generate-boot $(RELEASE)

Generating a release package

The final form of a release is a tarball containing all the files required to run our program, and again systools knows how to generate it. So let us write another Erlang program named generate-tarball:

#!/usr/bin/env escript

main([ReleaseName]) ->
  ERTSPath = code:root_dir(),
  Options = [no_warn_sasl, {path, ["ebin"]}, {erts, ERTSPath}, silent],
  case systools:make_tar(ReleaseName, Options) of
    {ok, _Module, []} ->
      ok;
    {ok, Module, Warnings} ->
      log_error("warning: ~ts", [Module:format_warning(Warnings)]);
    {error, Module, Errors} ->
      log_error("error: ~ts", [Module:format_error(Errors)]),
      halt(1)
  end;
main(_Args) ->
  log_error("usage: ~ts <release-name>~n", [escript:script_name()]),
  halt(1).

log_error(Format, Args) ->
  io:format(standard_error, Format, Args).

The code is very similar to the one we wrote to generate the boot script. We introduce a new option, {erts, ERTSPath}, to tell systools to add the Erlang runtime to the release. This way, our release will be fully standalone: we will be able to run it anywhere without having to install Erlang. This option requires the path containing Erlang libraries, and we obtain it with the code:root_dir function.

Then we add the new release target to the Makefile:

release: $(RELEASE).tar.gz

$(RELEASE).tar.gz: $(ERL_OBJ) $(RELEASE).boot
	./generate-tarball $(RELEASE)

At this point, we can simply run make release, and let GNU Make create the archive.

Running the application

Now that we have a tarball, we can extract it somewhere and start Erlang:

mkdir /tmp/hello
tar -xf hello.tar.gz -C /tmp/hello
cd /tmp/hello
./erts-13.2/bin/erl -noshell -boot releases/1.0.0/start

As you can see, the release contains Erlang itself. We start it with our boot script, which has been stored in releases/1.0.0/start.boot, and our application starts.

Conclusion

We can now build an Erlang release from scratch! Obviously a real world application would have other requirements such as handling multiple applications (internal or added from external dependencies), tests, several build profiles, extra data files or scripts to be added the release… And while it can be handled with systools and a few scripts, you probably want to start looking at either Rebar3 or Erlang.mk.

But you should now have a better understanding of the whole build and release process. As usual, the best way to figure out how something works is to get rid of the high level tools and try to do it on your own.

Share the word!

Liked my article? Follow me on Twitter or on Mastodon to see what I'm up to.