While setting up my dev environment to play around with golang I came across an issue when downloading packages using the go get package-name command

~# go get github.com/tools/godep
# cd .; git clone https://github.com/tools/godep /go/src/github.com/tools/godep
Cloning into '/go/src/github.com/tools/godep'...  
fatal: unable to access 'https://github.com/tools/godep/': Problem with the SSL CA cert (path? access rights?)  
package github.com/tools/godep: exit status 128  

At first I thought it was something wrong with my boot2docker VM since I had disabled TLS after encountering issues when doing upgrades with the certificates getting corrupted.
However, after chatting with some people at the #go-nuts IRC channel somebody suggested I could be under a MITM attack. I used Steve Gibson HTTPs fingerprint service and confirmed that nobody was attacking me.

To narrow down the possible causes of the issue I used curl to get github's finger print from the following locations

  • mac
  • boot2docker
  • official ubuntu image
  • customized ubuntu image

Below are the results:

mac

Connected to github.com (192.30.252.130) port 443 (#0)  
TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256  
Server certificate: github.com  
Server certificate: DigiCert SHA2 Extended Validation Server CA  
Server certificate: DigiCert High Assurance EV Root CA  

boot2docker

Connected to github.com (192.30.252.128) port 443 (#0)  
successfully set certificate verify locations: CAfile: /usr/local/etc/ssl/certs/ca-certificates.crt  
CApath: none  
SSLv3, TLS handshake, Client hello (1):  
SSLv3, TLS handshake, Server hello (2):  
SSLv3, TLS handshake, CERT (11):  
SSLv3, TLS handshake, Server key exchange (12):  
SSLv3, TLS handshake, Server finished (14):  
SSLv3, TLS handshake, Client key exchange (16):  
SSLv3, TLS change cipher, Client hello (1):  
SSLv3, TLS handshake, Finished (20):  
SSLv3, TLS change cipher, Client hello (1):  
SSLv3, TLS handshake, Finished (20):  
SSL connection using ECDHE-RSA-AES128-SHA  
Server certificate:  
subject: businessCategory=Private Organization; 1.3.6.1.4.1.311.60.2.1.3=US; 1.3.6.1.4.1.311.60.2.1.2=Delaware; serialNumber=5157550; street=548 4th Street; postalCode=94107; C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=github.com  
start date: 2014-04-08 00:00:00 GMT  
expire date: 2016-04-12 12:00:00 GMT  
subjectAltName: github.com matched  
issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA  
SSL certificate verify ok.  

Official Ubuntu Image

Hostname was NOT found in DNS cache  
Trying 192.30.252.128...  
Connected to github.com (192.30.252.128) port 443 (#0)  
successfully set certificate verify locations:  
CAfile: none  
CApath: /etc/ssl/certs  
SSLv3, TLS handshake, Client hello (1):  
SSLv3, TLS handshake, Server hello (2):  
SSLv3, TLS handshake, CERT (11):  
SSLv3, TLS alert, Server hello (2):  
SSL certificate problem: unable to get local issuer certificate  
Closing connection 0  

Customized Ubuntu Image

Connected to github.com (192.30.252.128) port 443 (#0)  
successfully set certificate verify locations:  
CAfile: none  
CApath: /etc/ssl/certs  
SSLv3, TLS handshake, Client hello (1):  
SSLv3, TLS handshake, Server hello (2):  
SSLv3, TLS handshake, CERT (11):  
SSLv3, TLS alert, Server hello (2):  
SSL certificate problem: unable to get local issuer certificate  
Closing connection 0  
curl: (60) SSL certificate problem: unable to get local issuer certificate  
More details here: http://curl.haxx.se/docs/sslcerts.html  

Root cause of the problems

After checking the results it was clear that something I had done while customizing my Ubuntu image was the source of the CA store problems.
The Dockerfile for my custom Ubuntu image was the following:

FROM ubuntu:14.04.3

RUN apt-get update

RUN apt-get install -y \  
    curl git vim build-essential \
    —no-install-recommends 

.....

The culprit was the --no-install-recommends.

I first saw the --no-install recommends flag being used in the Dockerfile for the golang official image

FROM buildpack-deps:jessie-scm

# gcc for cgo
RUN apt-get update && apt-get install -y \  
        gcc libc6-dev make \
        --no-install-recommends \
    && rm -rf /var/lib/apt/lists/*

ENV GOLANG_VERSION 1.4.2

RUN curl -sSL https://golang.org/dl/go$GOLANG_VERSION.src.tar.gz \  
        | tar -v -C /usr/src -xz

RUN cd /usr/src/go/src && ./make.bash --no-clean 2>&1

ENV PATH /usr/src/go/bin:$PATH

RUN mkdir -p /go/src /go/bin && chmod -R 777 /go  
ENV GOPATH /go  
ENV PATH /go/bin:$PATH  
WORKDIR /go

COPY go-wrapper /usr/local/bin/  

The flag tells apt-get to only install the required packages and drop the recommended packages.
When installing packages through apt-get there are three categories

  • required
  • recommended
  • suggested

By default the suggested packages are not installed and the recommended and required are.
The flags are the following

  • --no-install-recommends -- Do not consider recommended packages as a dependency for installing. Configuration Item: APT::Install-Recommends.
  • --install-suggests -- Consider suggested packages as a dependency for installing. Configuration Item: APT::Install-Suggests.

Checking the package dependency for curl I found that the package that installs the CA root certificates was called ca-certificates and it was listed as recommended

curl package list

  • The following extra packages will be installed:
    • libasn1-8-heimdal libcurl3 libgssapi-krb5-2 libgssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal libheimntlm0-heimdal libhx509-5-heimdal libidn11 libk5crypto3 libkeyutils1 libkrb5-26-heimdal libkrb5-3 libkrb5support0 libldap-2.4-2 libroken18-heimdal librtmp0 libsasl2-2 libsasl2-modules-db libwind0-heimdal
  • Suggested packages:
    • krb5-doc krb5-user
  • Recommended packages:
    • ca-certificates krb5-locales libsasl2-modules

So to solve the issue I simply removed the --no-install-recommends flag from my custom Ubuntu image Dockerfile and I was able to download packages using go get