Happy Gopher on CircleCI

CircleCI

CircleCI is one of the best CI providers nowadays. It has many features that many users love, what features? please go read all the stuffs. For freebies hunter, in addition of 3 free containers for public projects, it is also have one container free for private projects, as long as you… ahh, Okay, I will not talking about marketing stuff here, just go ahead to https://circleci.com and happy link surfing!. Oh, I will be waiting for you, don’t worry.

Okay, you’ve already came back? Great, now let’s continue!

Let’s say we have a brand new Go library, a very small library actually, the library usually will looks like:

$ tree
.
├── README.md
├── hello.go
└── hello_test.go
0 directories, 3 files

With this typical library structure, CircleCI is automatically detect the project type and run proper tests when you add your library repository as CircleCI project. Yes, no configuration needed. It just works!

### CIRCLECI BUILD OUTPUT ###
$ go get -t -d -v ./...
$ go build -v
$ go test -v -race ./...
=== RUN   TestSay
--- PASS: TestSay (0.00s)
PASS
ok  	github.com/subosito/hello-circle	1.008s

Dependencies? Vendor!

To improve the ability of your library, a decision is made to add some external libraries to it and use vendor tool, like govendor to manage the dependencies. Now, library directories will looks like:

$ tree -L 4
.
├── README.md
├── hello.go
├── hello_test.go
└── vendor
    ├── bitbucket.org
    │   └── pkg
    │       └── inflect
    ├── github.com
    │   ├── davecgh
    │   │   └── go-spew
    │   ├── pmezard
    │   │   └── go-difflib
    │   └── stretchr
    │       └── testify
    └── vendor.json
11 directories, 4 files

Does CircleCI automatically support dependencies on vendor directory? Oops, unfortunately not.

### CIRCLECI BUILD OUTPUT ###
$ go get -t -d -v ./...
$ go build -v
bitbucket.org/pkg/inflect
github.com/subosito/hello-circle

Ohh, that’s because CircleCI by default using old Go version isn’t? yeah, correct!. When you add project to CircleCI, it’s automatically use:

CIRCLE_BUILD_IMAGE=ubuntu-12.04

According to the CircleCI documentation about Ubuntu 12.04 environments. It’s still using Go 1.5.3. So vendor directory is not supported by default. There are two ways for handle this issue:

  1. Keep using Ubuntu 12.04 and set GO15VENDOREXPERIMENT=1**** environment variable
  2. Use newer image Ubuntu 14.04 which has Go 1.6.2 installed

Let’s play with each option :)

Ubuntu 12.04 with GO15VENDOREXPERIMENT=1

To add GO15VENDOREXPERIMENT=1 environment variable you can create custom CircleCI configuration, called circle.yml:

# circle.yml
machine:
  environment:
    GO15VENDOREXPERIMENT: 1

Now, CircleCI is using dependencies on vendor directory instead of remote ones:

### CIRCLECI BUILD OUTPUT ###
$ go get -t -d -v ./...
$ go build -v
github.com/subosito/hello-circle/vendor/bitbucket.org/pkg/inflect
github.com/subosito/hello-circle

But, but, CircleCI looks like still fetching perform go get to fetch dependencies although it’s already on vendor directory. Yeah, let’s try to fix that.

# circle.yml
machine:
  environment:
    GO15VENDOREXPERIMENT: 1
dependencies:
  override:
    - go get -u github.com/kardianos/govendor
    - go build -v

and.. build output from CircleCI is, uh oh, FAILED:

### CIRCLECI BUILD OUTPUT ###
$ go test -v -race ./...
# _/home/ubuntu/hello-circle
hello_test.go:7:2: cannot find package "github.com/subosito/hello-circle" in any of:
	/usr/local/go/src/github.com/subosito/hello-circle (from $GOROOT)
	/home/ubuntu/.go_workspace/src/github.com/subosito/hello-circle (from $GOPATH)
	/usr/local/go_workspace/src/github.com/subosito/hello-circle
FAIL	_/home/ubuntu/hello-circle [setup failed]

CircleCI is not able to setup Go environment, which basically set GOPATH, make symbolic links on proper location, etc. So, it’s broken. Broken dude.

Ok, let’s change configuration a bit. Seems override dependencies makes CircleCI failed to setting up Go environment properly. It’s a bug that CircleCI needs to dig through.

Here’s the updated circle.yml:

# circle.yml
machine:
  environment:
    GO15VENDOREXPERIMENT: 1
dependencies:
  pre:
    - go get -u github.com/kardianos/govendor

Well, because our attempt to remove default go get failed, we can skip that for later. Let’s take a look at another area which can be improved, which is test runner.

### CIRCLECI BUILD OUTPUT ###
$ go test -v -race ./...
=== RUN   TestSay
--- PASS: TestSay (0.00s)
PASS
ok  	github.com/subosito/hello-circle	1.012s
?   	github.com/subosito/hello-circle/vendor/bitbucket.org/pkg/inflect	[no test files]
?   	github.com/subosito/hello-circle/vendor/github.com/davecgh/go-spew/spew	[no test files]
?   	github.com/subosito/hello-circle/vendor/github.com/pmezard/go-difflib/difflib	[no test files]
?   	github.com/subosito/hello-circle/vendor/github.com/stretchr/testify/assert	[no test files]

By default, CircleCI run the tests recursively. So, any package that lives on vendor directory also tried to be tested. To change the behaviour, we can tweak circle.yml again:

# circle.yml
machine:
  environment:
    GO15VENDOREXPERIMENT: 1
dependencies:
  pre:
    - go get -u github.com/kardianos/govendor
test:
  override:
    - govendor test -v +local

Now, CircleCI build output is great again! eh.

### CIRCLECI BUILD OUTPUT ###
$ govendor test -v +local
=== RUN   TestSay
--- PASS: TestSay (0.00s)
PASS
ok  	github.com/subosito/hello-circle	0.005s

We are done for Ubuntu 12.04, let’s handle Ubuntu 14.04.

Ubuntu 14.04 with Go 1.6

There is only few steps involved for this. First, change default OS for build in Project settings > Build environments.

CircleCI OS setting

Second, remove unused environment variable GO15VENDOREXPERIMENT from circle.yml:

# circle.yml
dependencies:
  pre:
    - go get -u github.com/kardianos/govendor
test:
  override:
    - govendor test -v +local

That’s it.

### CIRCLECI BUILD OUTPUT ###
# CIRCLE_BUILD_IMAGE=ubuntu-14.04
$ govendor test -v +local
=== RUN   TestSay
--- PASS: TestSay (0.00s)
PASS
ok  	github.com/subosito/hello-circle	0.004s

Coverage Reports? Easy!

Enabling coverage reports is easy, it’s just matter of passing additional flag -coverprofile=coverage.out, like:

# circle.yml (stripped)
test:
  override:
    - govendor test -v -coverprofile=coverage.out +local

On CircleCI the output will looks like:

### CIRCLECI BUILD OUTPUT ###
$ govendor test -v -coverprofile=coverage.out +local
=== RUN   TestSay
--- PASS: TestSay (0.00s)
PASS
coverage: 100.0% of statements
ok  	github.com/subosito/hello-circle	0.004s

Multiple Packages

Since a new package added, our library now has multiple packages on it, so we need to restructure files to accommodates it:

$ tree -I coverage.out -L 4
.
├── README.md
├── circle.yml
├── hello
│   ├── hello.go
│   └── hello_test.go
├── howdy
│   ├── howdy.go
│   └── howdy_test.go
└── vendor
    ├── bitbucket.org
    │   └── pkg
    │       └── inflect
    ├── github.com
    │   ├── davecgh
    │   │   └── go-spew
    │   ├── pmezard
    │   │   └── go-difflib
    │   └── stretchr
    │       └── testify
    └── vendor.json
13 directories, 7 files

Oops, looks like govendor’s coverage reports is not working using new directory structure, we get errors like:

### LOCAL MACHINE ###
$ govendor test -v -coverprofile=coverage.out +local
cannot use test profile flag with multiple packages
Error: exit status 1

### CIRCLECI MACHINE ###
$ govendor test -v -coverprofile=coverage.out +local
Error: Package "/home/ubuntu/hello-circle" not a go package or not in GOPATH.
govendor test -v -coverprofile=coverage.out +local returned exit code 2

There are two issues here:

  • First, Govendor cover doesn’t work on multiple packages. To handle this we need to create a custom script so it can works properly. Let’s create coverage.sh:
#!/bin/bash

echo "mode: count" > coverage.out

PACKAGES=`govendor list -no-status +local`
EXIT_CODE=0

for PKG in $PACKAGES; do
  echo =-= $PKG

  govendor test -v -coverprofile=profile.out -covermode=count $PKG; __EXIT_CODE__=$?

  if [ "$__EXIT_CODE__" -ne "0" ]; then
    EXIT_CODE=$__EXIT_CODE__
  fi

  if [ -f profile.out ]; then
    tail -n +2 profile.out >> coverage.out; rm profile.out
  fi
done

exit $EXIT_CODE

Note: We change covermode to count so we can know how many times each statement run. We also store EXIT CODE so, if a package failed, then it will return exit code properly, in other words, it marks tests as failed.

  • Second, CircleCI expects a repository is a single package, multiple packages within a single repository is not supported yet. That’s why while it’s working on local machine, it doesn’t work on CircleCI.
### LOCAL MACHINE ###
$ ./coverage.sh
=-= github.com/subosito/hello-circle/hello
=== RUN   TestSay
--- PASS: TestSay (0.00s)
PASS
coverage: 100.0% of statements
ok      github.com/subosito/hello-circle/hello  0.013s
=-= github.com/subosito/hello-circle/howdy
=== RUN   TestHi
--- PASS: TestHi (0.00s)
PASS
coverage: 100.0% of statements
ok      github.com/subosito/hello-circle/howdy  0.012s

### CIRCLECI MACHINE ###
$ ./coverage.sh
Error: Package "/home/ubuntu/hello-circle" not a go package or not in GOPATH.

The issue happen like what we have earlier when we tried to override dependencies. CircleCI unable to set up Go environment.

So, now we have conclusion that both overriding dependencies configuration and having multiple packages in a single repository will breaks CircleCI Go environment setup.


Manual Environment Setup

The only way for tackling the issue is setting up Go environment manually. We can do that via circle.yml. The best way of doing this is replicate of what CircleCI default setup done as described here.

CircleCI places all projects in the ubuntu user’s home directory as /home/ubuntu/<REPO_NAME>. To work with Go’s expected directory structure, a symlink is placed to your project’s directory at /home/ubuntu/.go_project/src/github.com/<USER>/<REPO_NAME>.

It looks like:

machine:
  environment:
    GOPATH: "${HOME}/.go_workspace:/usr/local/go_workspace:${HOME}/.go_project"
dependencies:
  pre:
    - go get -u github.com/kardianos/govendor
  override:
    - mkdir -p "${HOME}/.go_project/src/github.com/${CIRCLE_PROJECT_USERNAME}"
    - ln -sf "${HOME}/${CIRCLE_PROJECT_REPONAME}/" "${HOME}/.go_project/src/github.com/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}"
test:
  override:
    - cd "${PROJECT_PATH}" && ./coverage.sh

Unfortunately, default setup is not working fine with our coverage script. On coverage script we use govendor list to get local package names. Using symbolic link, the command returns nothing.

$ govendor list -no-status +local

To fix that issue, we can do rsync instead of make symbolic link. Also our configuration contains some duplications, lets fix that as well:

machine:
  environment:
    PROJECT_GOPATH: "${HOME}/.go_project"
    PROJECT_PARENT_PATH: "${PROJECT_GOPATH}/src/github.com/${CIRCLE_PROJECT_USERNAME}"
    PROJECT_PATH: "${PROJECT_PARENT_PATH}/${CIRCLE_PROJECT_REPONAME}"
    GOPATH: "${HOME}/.go_workspace:/usr/local/go_workspace:${PROJECT_GOPATH}"
dependencies:
  pre:
    - go get -u github.com/kardianos/govendor
  override:
    - mkdir -p "${PROJECT_PARENT_PATH}"
    - rsync -avC "${HOME}/${CIRCLE_PROJECT_REPONAME}/" "${PROJECT_PATH}"
test:
  override:
    - cd "${PROJECT_PATH}" && ./coverage.sh

Now, CircleCI build is green again!

### CIRCLECI BUILD OUTPUT ###
$ go get -u github.com/kardianos/govendor
$ mkdir -p "${PROJECT_PARENT_PATH}"
$ rsync -avC "${HOME}/${CIRCLE_PROJECT_REPONAME}/" "${PROJECT_PATH}"
$ cd "${PROJECT_PATH}" && ./coverage.sh
=-= github.com/subosito/hello-circle/hello
=== RUN   TestSay
--- PASS: TestSay (0.00s)
PASS
coverage: 100.0% of statements
ok  	github.com/subosito/hello-circle/hello	0.004s
=-= github.com/subosito/hello-circle/howdy
=== RUN   TestHi
--- PASS: TestHi (0.00s)
PASS
coverage: 100.0% of statements
ok  	github.com/subosito/hello-circle/howdy	0.004s

Coverage report for multiple packages is coverage.out. You can use code coverage viewer service to parse it. A good one is codecov, which you can integrate easily on CircleCI as well.

# circle.yml (trimmed)
test:
  override:
    - cd "${PROJECT_PATH}" && make coverage
  post:
    - cd "${PROJECT_PATH}" && bash <(curl -s https://codecov.io/bash) -f coverage.out

Happy testing! Be happy Gopher!