Nicolas Martyanoff – Brain dump

Taking control of your Go module paths

nicolas@n16f.net (Nicolas Martyanoff) — Sun, 07 Jul 2024 18:00:00 +0000

The canonical name of a Go module is its path. For third party modules, that means an URI. It is no secret that GitHub is currently the dominant code hosting platform, hence the fact that most Go modules are named github.com/<account>/<repository>. While I currently use GitHub for most of my repositories, I also want the ability to move out of it in the future. Fortunately the Go module system allows package authors to control how the package is accessed. One can use it to provide Go modules with their own domain and total control on module paths. Let us see how to do that.

How packages are downloaded

When the Go client downloads a module (either directly with GOPROXY=direct or when a proxy does it), it fetches the module URI with an HTTP request and a go-get=1 query parameter. The server includes a <meta> HTML element named go-import in the response, its value indicating how to actually download the module.

Let us check ourselves with my go-uuid module:

$ curl -s https://github.com/galdor/go-uuid\?go-get\=1 | grep go-import
  <meta name="go-import" content="github.com/galdor/go-uuid git https://github.com/galdor/go-uuid.git">

GitHub indicates that to download the github.com/galdor/go-uuid module, the client must clone the Git repository at https://github.com/galdor/go-uuid.git.

So for a module to be named example.com/foo, all you need is an HTTP server responding to requests at https://example.com/foo?go-get=1.

Using GitHub Pages

I wanted all my packages to be available at go.n16f.net/<name>. It is short, it looks good, and it opens the possibility to move the repositories out of GitHub in the future.

While I could have used my own HTTP server, I was worried about what would happen if someone needed to download a module when the server is down. Since I already use GitHub to host most repositories, I decided to use GitHub Pages: after all, what I need is a way to host HTML pages with the right <meta> tag at the right URI.

Generating repository pages

I created a GitHub repository for these pages. It was tempting to use a simple shell script to generate pages but I really did not want to use M4 and Ruby lets me use ERB templates, so Ruby it is.

All my Go modules are listed in a text file, and a small Ruby script generates both an index (not required but why not?) and a page for each module. The <meta> tag of each module page indicates that go.n16f.net/<name> can be accessed by cloning the Git repository at https://github.com/galdor/go-<name>. It is that simple.

Configuring the domain

Using your own domain means configuring it to point to GitHub pages.

First head to your domain name registrar website and configure your domain to point to GitHub pages. In my case, that means creating a CNAME DNS record named go.n16f.net. pointing to galdor.github.io.. You will notice that the GitHub Pages address is global to your account and not specific to each repository: GitHub takes care of routing, based on which domain is associated with each repository.

Then go to the GitHub Pages settings page and add your domain. You will have to create a TXT DNS record with a specific name and value to prove ownership of the domain.

Once done, go to the GitHub Pages settings page of the repository. Select the “GitHub Actions” source, and add the domain to “Custom domain”. Make sure to tick “Enforce HTTPS” while you are here.

Publishing pages

To generate and publish pages, we use a small GitHub Actions workflow. When new commits are pushed to the master branch, we run the script (the ubuntu-latest image already contains Ruby, so there is nothing to setup) and use official GitHub actions to upload and deploy the pages we just generated.

A quick check shows that the new URIs work as expected:

$ curl -s https://go.n16f.net/uuid\?go-get\=1 | grep go-import
    <meta name="go-import" content="go.n16f.net/uuid git https://github.com/galdor/go-uuid.git">

Updating modules

You can now update your Go modules: update the path in the go.mod file, and all import paths in source files. Yes this is annoying, and I would love a way to declare the path of each dependencies at project level, but Sed makes the task trivial.

Updating dependencies is a bit trickier. If your project depends on github.com/example/foo which you just renamed to example.com/foo, update all module paths in source files then run:

GOPROXY=direct go get example.com/foo@latest && go mod tidy

This will force Go to fetch the module directly from its source and not from the global proxy which may not be up-to-date yet.

Result

For a surprisingly small amount of work, I now have all my Go modules available at a path I fully control and I can add new modules by modifying a text file and letting the GitHub workflow handles the rest. And while the current setup is still dependent on GitHub, I can move it somewhere else without any change to module paths.

What is your excuse for not doing the same thing?

Golang workspaces, at last!

nicolas@n16f.net (Nicolas Martyanoff) — Tue, 28 Mar 2023 18:00:00 +0000

Workspaces are a feature introduced in March 2022 with Go 1.18. They did not really get a lot of publicity, and I have not had the chance to experiment with them until recently. However I am really glad I did because they improve a major aspect of my workflow: dealing with multiple modules.

Go modules are not ideal

Go modules were a big help when they were introduced in 2018, but they were always limited. Larger products are often split into multiple projects: one or more applications and several libraries and tools, meaning multiple modules.

While module versioning makes sure that components use the right version of each dependency, it is also really annoying during development. If your application foo depends on the go-bar library, you will often have to work on go-bar while testing the changes in foo. This means updating go-bar, commiting changes, pushing them, and updating the foo module with go get. Quite cumbersome.

A first improvement is the module replacement system. In the example above, we could instruct Go to use the local copy of go-bar when building foo. Assuming that foo and go-bar are at the same level in the filesystem:

go mod edit -replace example.com/myproject/go-bar=../go-bar

With this simple change, you have made your life easier: you can work on changes in go-bar, and immediately build and run foo without any additional operations. When you are done, you can still commit and push normally in go-bar.

But it is still not perfect. Module replacements are stored in the go.mod file, meaning that these changes will be pushed to your central repository circumventing dependency versioning and causing issues for other developers and CI processes. While module replacement works fine to point a module to a fork, it is not really a solution for our problem.

Using workspaces

Go workspaces let you create environments where you control the source of the modules you use, without having to modify these modules.

Going back to our example, we can solve our problem by creating a workspace for foo in which go-bar refers to the local copy. In the directory of foo:

go work init
go work use .
go work use ../go-bar

Nothing complicated here. The init subcommand creates the go.work file which will contain the configuration of the workspace. Then the use subcommand is called to add two modules to the workspace: the foo module in the current repository, i.e. ., and the one in the go-bar sibling directory.

At this point, building foo will correctly use the local copy of go-bar without having to modify any of the modules. Problem solved.

Even better, you can include replace declarations in the go.work file the exact same way as in go.mod file. These declarations will override those in module files, giving you total control on the environment, again without having to alter modules.

Committing the workspace file

Whether you commit the workspace file or not depends on your situation. When working in a mono-repository with other developers, committing a workspace file allows everyone to build the project the exact same way, using dependencies in the repository. It can also be practical with multiple repositories as long as you expect everyone to organize their local copies the same way.

But you can also keep your own workspace files without committing them. This gives you the ability to quickly switch to a local copy for a dependency or to replace a module by another one during development. For example this what I do for my go-raft library project. The program in the cmd/kvstore directory is based on the go-service library. During development, I sometimes have to add code to go-service. Therefore I have a workspace file in go-raft which references the local copy of go-service. But I do not commit it, to avoid affecting anyone trying to build go-raft.

This flexibility is really practical, and I’m quite satisfied with workspaces at this point.

What could be better

There is a small issue to keep in mind. If you get used to work with workspaces, committing your local dependencies and having your program use them automatically, you may still have to maintain module dependencies. It is not necessarily a problem if you commit the workspace file and always use it, but it makes sense to keep dependencies up-to-date in your module files.

The go work command has a sync subcommand whose description let me think that it would update modules included in the workspace to the right dependency versions, but it turns out not do to so when tracking pseudo-versions (i.e. when you are tracking a branch and not a specific tag).

Therefore I have to continue to update non-tagged dependencies with this usual (and quite excessive) command, here for go-service:

GOPROXY=direct go get github.com/galdor/go-service@latest
go mod tidy

This way I make sure to bypass the Go proxy, fetch the last version of the master branch and clean up the dependency list.

Despite this small inconvenience, Go workspaces have made my daily work much easier. Still a win!