A Primer To Go Modules
Traveling outside GOPATH
Go Modules was introduced in 2018 with Go version 1.11 and with the current (25th of February 2020) version 1.14 is considered ready for production use. It is fairly straight-forward and easy to use. This blog post gives some background on the historical transition to modules, what modules are and the advantages of using it.
What are Go Packages?
Modules use the existing package management of Go without changing it fundamentally. To better understand them it is therefore necessary to first understand what packages are.
An important and fundamental part of high-quality software is code reuse embodied in the principle “Don’t Repeat Yourself” — the DRY principle. You should never repeat the same code over and over again, but instead allow it to be reused as much as possible. This also increases the maintainability of code.
Packages help in organizing related Go source files together into an atomic unit. A package bundles functions and variables together and makes them available to the rest of a bigger project or to completely separate projects. This makes it easier to share code with other projects by allowing them to import and reuse it.
In this way Go encourages you to write small pieces of software components through packages, and compose your applications with these small packages.
Using packages has the following advantages:
- No name conflicts, since functions can have the same name while belonging to different packages.
- Allows reusing and sharing code among projects.
- Organizes related code together.
- Reduces compile times as packages are only re-compiled if their code is changed.
Declaring a package
A package is not much more than a directory inside your Go workspace with one or more Go source files, or other Go packages.
Every Go source file must belong to a package, which is declared at the top of a file using the following syntax:
All functions and variables defined in this source file then become part of the declared package.
You can export a member of your package by letting its declared name begin with a capital letter. Only then can other packages import and use it.
To import packages in Go use the
import ( "fmt" "time" "math" "math/rand" )
The last element of the import path is the package name available for your project. For example, the package imported with
math/rand is simply
rand and its exported function
Intn(n int) can be invoked with e.g.
rand.Intn(100). The package is imported with
math/rand because it is nested inside the
math package as a subdirectory.
It is possible to nest a package inside another by creating a subdirectory.
$GOPATH └── github.com └── bjarnemagnussen └── myproject ├── numbers # Some Package └── strings # Another Package └── greet # Nested Package └── texts.go
All import paths for non-builtin packages are relative to its
import ( "github.com/bjarnemagnussen/myproject/numbers" "github.com/bjarnemagnussen/myproject/strings" "github.com/bjarnemagnussen/myproject/strings/greet" )
Getting 3rd Party Packages
This makes it very easy to import third party packages. You can use the
go get command to download third party packages from remote repositories.
$ go get -u github.com/jinzhu/gorm
The above command fetches the
gorm package from Github and automatically stores it inside
GOPATH. To use it import the package to any of the source files in your project with:
What happens if different packages have the same names? Package alias can resolve naming conflicts between packages, or give a short name to an imported package.
import ( str "strings" // Package Alias "str" )
Vendoring And Versioning
Go packages do not provide any builtin versioning system. The Go FAQ suggested to create a local copy of a third party package that should stay frozen:
If you’re using an externally supplied package and worry that it might change in unexpected ways, the simplest solution is to copy it to your local repository. (This is the approach Google takes internally.) Store the copy under a new import path that identifies it as a local copy.
Over the years many different tools where proposed and developed around the idea of “vendoring” such as
dep. Around 2016 a discussion arose
to tag versions onto packages and prepare for a time when it would become valuable. The community agreed on
git release tags with semantic versioning.
Semantic versioning divides releases into
- a major (breaking) version,
- a minor (feature) version,
- and a patch version.
The full version is prepended with a single “v” and looks like “v1.2.3”, which translates into a major version 1, with minor version 2 and patch version 3. When there is a bugfix the patch version is increased. If features are added that are backwards compatible the minor version is increased. For changes to the code that remove backwards compatibility the major version is increased.
Version tags are used by vendoring tools such as
dep to define which commit state of a package to vendor. However, another approach was introduced in 2018 to completely eliminate vendoring and allow for project-based workflows instead of
GOPATH: Go Modules.
Today Go modules has replaced
dep and followed up on the work of yet another dependency manager called
vgo. Modules is a way of packaging software and is Go’s new simplistic dependency management that even allows for reproducible builds.
Imagine you are developing a Go program with a number of dependencies such as package
A. At the time of writing your program, package
A works in a set way.
However, what happens when the maintainers of package
A update their program to fix a bug or extend functionality? You might get lucky and their changes might not impact your application. However, you might be unlucky and these changes break your application.
This is where Go Modules comes in to save the day. By using modules, we can select the precise versions of a dependency that we wish to use and ensure that whenever we build our program, it always uses the specified versions.
A module is just a collection of Go packages stored in a directory with a
go.mod file at its root. As an example the following
go.mod file defines the module
module github.com/my/thing require ( github.com/some/dependency v1.2.3 github.com/another/dependency/v4 v4.0.0 )
There are four directives:
exclude. I refer to the excellent
Go Wiki page to read about the last two directives
require directive specifies the versions of the named packages that are used by this module as dependencies and will be revisited in more detail later.
A module path is defined with the
module directive. All packages in a module share the module path as a common prefix to their import paths. In other words, the module path and relative path from the
go.mod file to a package’s directory decides its import path.
For example, if you are creating a module for a repository
github.com/my/thing with two packages
bar, then the first line of the
go.mod file would declare the module path with
module github.com/my/thing. The corresponding directory structure could be:
thing ├── bar | └── bar.go ├── foo | └── foo.go └── go.mod
The import paths for the
bar packages would then be
github.com/my/thing/bar. Hence the package
foo would be imported in a Go source file with:
GOPATHfrom another module can be easily achieved using a VCS (e.g. github.com) for the dependency.
Go can automatically take care of downloading and importing a dependency if it is using a VCS, regardless of where its working directory is stored on disk. Alternatively, the
replace directive of
go.mod can be used to point to the dependency’s local directory, see
this answer on StackOverflow.
With Go Modules, adding a new dependency to any of your Go source files will have most Go commands (such as
go build and
go test) automatically download the highest version of that new package and add it as a direct dependency to the
go.mod file with the
For example, assume you create a new module from scratch with a
go.mod file only consisting of:
Adding a new import to a dependency
github.com/some/mod, which latest tagged release version is
v1.2.3, and running
go build ./... will automatically download the dependency and change your
go.mod file to reflect the new dependency with its latest version:
module github.com/my/thing require ( github.com/some/mod v1.2.3 )
github.com/some/mod is now a dependency with an allowed version greater than v1.2.3 but less than v2, since the assumed semantic version means that v2 is incompatible with v1.
Go Modules uses a minimal version selection for indirect dependencies. That means that Go Modules selects the highest of the versions explicitly listed by a
require directive in your module or any one of its dependencies. If your module depends on
A that itself has a
require D v1.0.0 and your module also depends on
B that has a
require D v1.1.1, then Go Modules would select
v1.1.1 of dependency
D. This selection of
v1.1.1 remains consistent even if some time later a
D becomes available. This is in contrast to
go dep, which always selects the highest compatible released version under the assumption of semantic versioning.
To see a list of the selected module versions (including indirect dependencies), you can use
go list -m all.
If later an indirect dependency is removed, Go modules will still keep track of the “latest not-greatest” version. In other words, if we were to remove from our module the dependency
B containing a
require D v1.1.1 but keep dependency
A with a
require D v1.0.0, then Go modules would not fallback to
v1.0.0 but instead keep
Keeping track of dependency versions is done with a
go.sum file is an opaque reliability meta data file and it should not be used to understand your dependencies. Any time your Go Module downloads a version of a dependency not yet seen, that version is tagged inside the
go.sum file with its corresponding git commit hash acting as a cryptographic checksum.
go.sum file retains checksums for dependency versions even after you stop using that particular dependency or version. This allows validation of the checksums if you later resume using something, providing additional safety. The
go.sum will therefore frequently have more dependency listed than your
go.sum file should always be committed along with your
go.sum has the following benefits:
- If someone clones your repository and downloads your dependencies, they will receive an error if there is any mismatch between their downloaded copies of your dependencies and the corresponding entries in your
- In addition,
go mod verifychecks that the on-disk cached copies of module downloads still match the entries in
Activating And Using Modules
Enough of the theory! Let’s see how we can add Go Modules to a project and use it.
One of the main features of modules is to work on Go projects outside of
GOPATH. We will therefore start by navigating to the root directory of some imaginary project not stored in the
$ cd <project path outside $GOPATH/src> # e.g., cd ~/projects/hello
- Create the initial module definition by automatically writing it to the
$ go mod init
go mod init will often be able to use auxiliary data (such as VCS meta-data) to automatically determine the appropriate module path. But if
go mod init should state that it cannot automatically determine the module path, you can supply the module path as an optional argument with
go mod init github.com/my/thing.
- Build the module with
$ go build ./...
Building the module will automatically download all necessary dependencies and choose their latest versions. The direct dependencies will show up with the
require directive inside
go.mod, while both the direct and indirect dependencies are tagged to their commit hashes inside
- It is good practice to run
go mod tidybefore committing code to your repository.
This will ensure that the
go.mod file contains the dependency requirements for all possible combinations of OS, architecture, and build tags, including for testing purposes. This will help others on your team and your CI/CD environments.
go mod tidy will also fix any inconsistency reflected by a missing
go.mod file of a dependency of your module, e.g. because the dependency has not yet opted in to modules itself, or if its
go.mod file is missing one or more of its dependencies, e.g. because the module author did not run
go mod tidy. The missing transitive dependencies will be added to your module’s requirements along with an
// indirect comment to indicate that the dependency is not from a direct import within your module.
This behaviour is how modules provide 100% reproducible builds and tests by recording precise dependency information.
To update a specific dependency inside your module and all its indirect dependency you run as usually
$ go get -u github/some/package
This will also automatically change both
go.sum to reflect the new version(s).
To upgrade all direct and indirect dependencies to their latest versions do the following:
- Upgrading to latest minor or patch releases:
go get -u ./...(and add -t to also upgrade test dependencies).
- Upgrading to latest patch releases only:
go get -u=patch ./...(and add -t to also upgrade test dependencies).
I may have skipped a lot of small details in this post. But the idea behind Go Modules is that it should often be possible to just magically work without configuration. However, knowing some background from where Go Modules comes helps in understanding the design choices. Today Go Modules should be used with every new project and can also easily be activated for existing ones.