A Primer To Go Modules

Go Modules was introduced in 2018 with Go version 1.11 and with the current (25th of February 2020) version 1.14 is considered ready for production use. It is fairly straight-forward and easy to use. This blog post gives some background on the historical transition to modules, what modules are and the advantages of using it.

What are Go Packages?

Modules use the existing package management of Go without changing it fundamentally. To better understand them it is therefore necessary to first understand what packages are.

An important and fundamental part of high-quality software is code reuse embodied in the principle “Don’t Repeat Yourself” — the DRY principle. You should never repeat the same code over and over again, but instead allow it to be reused as much as possible. This also increases the maintainability of code.

Packages help in organizing related Go source files together into an atomic unit. A package bundles functions and variables together and makes them available to the rest of a bigger project or to completely separate projects. This makes it easier to share code with other projects by allowing them to import and reuse it.

In this way Go encourages you to write small pieces of software components through packages, and compose your applications with these small packages.

Using packages has the following advantages:

• No name conflicts, since functions can have the same name while belonging to different packages.
• Allows reusing and sharing code among projects.
• Organizes related code together.
• Reduces compile times as packages are only re-compiled if their code is changed.

Declaring a package

A package is not much more than a directory inside your Go workspace with one or more Go source files, or other Go packages.

Every Go source file must belong to a package, which is declared at the top of a file using the following syntax:

package <packagename>

All functions and variables defined in this source file then become part of the declared package.

You can export a member of your package by letting its declared name begin with a capital letter. Only then can other packages import and use it.

Importing packages

To import packages in Go use the import syntax

import (
  "fmt"
  "time"
  "math"
  "math/rand"
)

The last element of the import path is the package name available for your project. For example, the package imported with math/rand is simply rand and its exported function Intn(n int) can be invoked with e.g. rand.Intn(100). The package is imported with math/rand because it is nested inside the math package as a subdirectory.

Nesting

It is possible to nest a package inside another by creating a subdirectory.

$GOPATH
└── github.com
    └── bjarnemagnussen
        └── myproject
            ├── numbers          # Some Package
            └── strings          # Another Package
                └── greet        # Nested Package
                    └── texts.go

The GOPATH

All import paths for non-builtin packages are relative to its GOPATH.

import (
  "github.com/bjarnemagnussen/myproject/numbers"
  "github.com/bjarnemagnussen/myproject/strings"  
  "github.com/bjarnemagnussen/myproject/strings/greet"
)

Getting 3rd Party Packages

This makes it very easy to import third party packages. You can use the go get command to download third party packages from remote repositories.

$ go get -u github.com/jinzhu/gorm

The above command fetches the gorm package from Github and automatically stores it inside GOPATH. To use it import the package to any of the source files in your project with:

import "github.com/jinzhu/gorm"

Package Alias

What happens if different packages have the same names? Package alias can resolve naming conflicts between packages, or give a short name to an imported package.

import (
  str "strings"   // Package Alias "str"
)

Vendoring And Versioning

Go packages do not provide any builtin versioning system. The Go FAQ suggested to create a local copy of a third party package that should stay frozen:

If you’re using an externally supplied package and worry that it might change in unexpected ways, the simplest solution is to copy it to your local repository. (This is the approach Google takes internally.) Store the copy under a new import path that identifies it as a local copy.

Over the years many different tools where proposed and developed around the idea of “vendoring” such as vend, gb and dep. Around 2016 a discussion arose to tag versions onto packages and prepare for a time when it would become valuable. The community agreed on git release tags with semantic versioning.

Semantic versioning divides releases into

• a major (breaking) version,
• a minor (feature) version,
• and a patch version.

The full version is prepended with a single “v” and looks like “v1.2.3”, which translates into a major version 1, with minor version 2 and patch version 3. When there is a bugfix the patch version is increased. If features are added that are backwards compatible the minor version is increased. For changes to the code that remove backwards compatibility the major version is increased.

Version tags are used by vendoring tools such as dep to define which commit state of a package to vendor. However, another approach was introduced in 2018 to completely eliminate vendoring and allow for project-based workflows instead of GOPATH: Go Modules.

Go Modules

Today Go modules has replaced dep and followed up on the work of yet another dependency manager called vgo. Modules is a way of packaging software and is Go’s new simplistic dependency management that even allows for reproducible builds.

Imagine you are developing a Go program with a number of dependencies such as package A. At the time of writing your program, package A works in a set way.

However, what happens when the maintainers of package A update their program to fix a bug or extend functionality? You might get lucky and their changes might not impact your application. However, you might be unlucky and these changes break your application.

This is where Go Modules comes in to save the day. By using modules, we can select the precise versions of a dependency that we wish to use and ensure that whenever we build our program, it always uses the specified versions.

go.mod

A module is just a collection of Go packages stored in a directory with a go.mod file at its root. As an example the following go.mod file defines the module github.com/my/thing:

module github.com/my/thing

require (
    github.com/some/dependency v1.2.3
    github.com/another/dependency/v4 v4.0.0
)

There are four directives: module, require, replace and exclude. I refer to the excellent Go Wiki page to read about the last two directives replace and exclude. The require directive specifies the versions of the named packages that are used by this module as dependencies and will be revisited in more detail later.

A module path is defined with the module directive. All packages in a module share the module path as a common prefix to their import paths. In other words, the module path and relative path from the go.mod file to a package’s directory decides its import path.

For example, if you are creating a module for a repository github.com/my/thing with two packages foo and bar, then the first line of the go.mod file would declare the module path with module github.com/my/thing. The corresponding directory structure could be:

thing
├── bar
|   └── bar.go
├── foo
|   └── foo.go
└── go.mod

The import paths for the foo and bar packages would then be github.com/my/thing/foo and github.com/my/thing/bar. Hence the package foo would be imported in a Go source file with:

import "github.com/my/thing/foo"

Go can automatically take care of downloading and importing a dependency if it is using a VCS, regardless of where its working directory is stored on disk. Alternatively, the replace directive of go.mod can be used to point to the dependency’s local directory, see this answer on StackOverflow.

Version Selection

With Go Modules, adding a new dependency to any of your Go source files will have most Go commands (such as go build and go test) automatically download the highest version of that new package and add it as a direct dependency to the go.mod file with the require directive.

For example, assume you create a new module from scratch with a go.mod file only consisting of:

module github.com/my/thing

Adding a new import to a dependency github.com/some/mod, which latest tagged release version is v1.2.3, and running go build ./... will automatically download the dependency and change your go.mod file to reflect the new dependency with its latest version:

module github.com/my/thing

require (
  github.com/some/mod v1.2.3
)

The module github.com/some/mod is now a dependency with an allowed version greater than v1.2.3 but less than v2, since the assumed semantic version means that v2 is incompatible with v1.

Go Modules uses a minimal version selection for indirect dependencies. That means that Go Modules selects the highest of the versions explicitly listed by a require directive in your module or any one of its dependencies. If your module depends on A that itself has a require D v1.0.0 and your module also depends on B that has a require D v1.1.1, then Go Modules would select v1.1.1 of dependency D. This selection of v1.1.1 remains consistent even if some time later a v1.2.0 of D becomes available. This is in contrast to go dep, which always selects the highest compatible released version under the assumption of semantic versioning.

To see a list of the selected module versions (including indirect dependencies), you can use go list -m all.

If later an indirect dependency is removed, Go modules will still keep track of the “latest not-greatest” version. In other words, if we were to remove from our module the dependency B containing a require D v1.1.1 but keep dependency A with a require D v1.0.0, then Go modules would not fallback to v1.0.0 but instead keep v1.1.1 of D.

Keeping track of dependency versions is done with a go.sum file.

go.sum

The go.sum file is an opaque reliability meta data file and it should not be used to understand your dependencies. Any time your Go Module downloads a version of a dependency not yet seen, that version is tagged inside the go.sum file with its corresponding git commit hash acting as a cryptographic checksum.

The go.sum file retains checksums for dependency versions even after you stop using that particular dependency or version. This allows validation of the checksums if you later resume using something, providing additional safety. The go.sum will therefore frequently have more dependency listed than your go.mod file.

Your module’s go.sum file should always be committed along with your go.mod file.

Committing go.sum has the following benefits:

• If someone clones your repository and downloads your dependencies, they will receive an error if there is any mismatch between their downloaded copies of your dependencies and the corresponding entries in your go.sum.
• In addition, go mod verify checks that the on-disk cached copies of module downloads still match the entries in go.sum.

Activating And Using Modules

Enough of the theory! Let’s see how we can add Go Modules to a project and use it.

One of the main features of modules is to work on Go projects outside of GOPATH. We will therefore start by navigating to the root directory of some imaginary project not stored in the GOPATH

$ cd <project path outside $GOPATH/src>         # e.g., cd ~/projects/hello

1. Create the initial module definition by automatically writing it to the go.mod file with:

$ go mod init

go mod init will often be able to use auxiliary data (such as VCS meta-data) to automatically determine the appropriate module path. But if go mod init should state that it cannot automatically determine the module path, you can supply the module path as an optional argument with go mod init github.com/my/thing.

2. Build the module with

$ go build ./...

Building the module will automatically download all necessary dependencies and choose their latest versions. The direct dependencies will show up with the require directive inside go.mod, while both the direct and indirect dependencies are tagged to their commit hashes inside go.sum.

3. It is good practice to run go mod tidy before committing code to your repository.

This will ensure that the go.mod file contains the dependency requirements for all possible combinations of OS, architecture, and build tags, including for testing purposes. This will help others on your team and your CI/CD environments.

Running go mod tidy will also fix any inconsistency reflected by a missing go.mod file of a dependency of your module, e.g. because the dependency has not yet opted in to modules itself, or if its go.mod file is missing one or more of its dependencies, e.g. because the module author did not run go mod tidy. The missing transitive dependencies will be added to your module’s requirements along with an // indirect comment to indicate that the dependency is not from a direct import within your module.

This behaviour is how modules provide 100% reproducible builds and tests by recording precise dependency information.

Updating Dependencies

To update a specific dependency inside your module and all its indirect dependency you run as usually

$ go get -u github/some/package

This will also automatically change both go.mod and go.sum to reflect the new version(s).

To upgrade all direct and indirect dependencies to their latest versions do the following:

• Upgrading to latest minor or patch releases: go get -u ./... (and add -t to also upgrade test dependencies).
• Upgrading to latest patch releases only: go get -u=patch ./... (and add -t to also upgrade test dependencies).

Conclusion

I may have skipped a lot of small details in this post. But the idea behind Go Modules is that it should often be possible to just magically work without configuration. However, knowing some background from where Go Modules comes helps in understanding the design choices. Today Go Modules should be used with every new project and can also easily be activated for existing ones.