docker

Running a local docker registry

Why would you want to run a local registry?

  • If you are using vagrant to mimic your qa/staging/prod environments and want to make sure you run the same host OS
  • If you want to test multi-host deployments with Rancher, Kubernetes, Mesos or any other orchestration framework

Without a local registry every time a new docker host is provisioned all the images need to be pulled from the remote registry, this might not be a huge issue, but depending on how many services you run it could take some time to re-download all the images. By having a local registry caching all the images it makes sure you can provision ready to use environments very quickly.

To run the registry there are a few steps

Create configuration file

version: 0.1  
log:  
  level: debug
  formatter: text
http:  
  secret: secret
  addr: :5000
storage:  
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
proxy:  
  remoteurl: https://registry-1.docker.io

It is possible to override the settings of the config file via environment variables by using the SECTION_SECTION_... format, eg; STORAGE_CACHE_FILESYSTEM=/path
However, it is recommended to drive the configuration of the registry via a file instead of environment variables.
Settings worth mentioning:

  • storage_filesystem_rootdirectory - where to storage the image layers and repositories
  • proxy - enables a registry to be configured as a pull through cache

The official docs describe the configuration settings in more details.

Create compose file

version: '2'  
services:  
  registry:
    image: registry:2.4
    ports:
      - "5000:5000"
      - "5001:5001"
    volumes:
      - /data/registry:/var/lib/registry
    command:
      serve /var/lib/registry/config.yml
    networks:
      default:
        aliases:
          - registry.local

Bring the registry up

docker-compose up -d  

Verify the registry is online

curl -I localhost:5000  

List the repositories

curl localhost:5000/v2/_catalog  

It should come back as an empty array

{
    "repositories": []
}

Configure docker daemon to use the local registry

Where to change the configuration will depend on the distro you are using, but in any case, make sure the docker daemon is started with the following attributes:

Pull MySQL image

docker pull mysql:latest  

Verify the image has been cached

curl localhost:5000/v2/_catalog  

The MySQL repo should be present in the list

{
    "repositories": [
        "library/mysql",
    ]
}

List the image tags

curl localhost:5000/v2/library/mysql/tags/list  

Should list all the available tags for the image

{
    "name": "library/mysql",
    "tags": [
        "5.5.40",
        "5.5.41",
        "5.5.42",
        "5.5.43",
        "5.5.44",
        "5.5.45",
        "5.5.46",
        "5.5.47",
        "5.5.48",
        "5.5.49",
        "5.5.50",
        "5.5.51",
        "5.5.52",
        "5.5",
        "5.6.17",
        "5.6.20",
        "5.6.21",
        "5.6.22",
        "5.6.23",
        "5.6.24",
        "5.6.25",
        "5.6.26",
        "5.6.27",
        "5.6.28",
        "5.6.29",
        "5.6.30",
        "5.6.31",
        "5.6.32",
        "5.6.33",
        "5.6",
        "5.7.10",
        "5.7.11",
        "5.7.12",
        "5.7.13",
        "5.7.14",
        "5.7.15",
        "5.7.4-m14",
        "5.7.4",
        "5.7.5-m15",
        "5.7.5",
        "5.7.6-m16",
        "5.7.6",
        "5.7.7-rc",
        "5.7.7",
        "5.7.8-rc",
        "5.7.8",
        "5.7.9",
        "5.7",
        "5",
        "latest"
    ]
}

Using NATS instead of HTTP for inter service communication

In the new micro service architecture the traditional monolithic application is broken down into multiple small scoped domains. That brings several benefits both from a development perspective since micro services can be iterated individually and also performance since they can be scaled separately.

Now with the move to a micro service architecture one common problem is how to make all the services communicate with one another?
Typically an application exposes a REST API interface to the client so naturally the first option is to start using HTTP.
The question then becomes, is HTTP the best messaging protocol for inter service communication?

Looking at the HTTP's definition from the official RFC

The Hypertext Transfer Protocol (HTTP) is an application-level  
protocol for distributed, collaborative, hypermedia information  
systems. It is a generic, stateless, protocol which can be used for  
many tasks beyond its use for hypertext, such as name servers and  
distributed object management systems, through extension of its  
request methods, error codes and headers.  

It is clear that HTTP wasn't built to provide a communication layer between micro services.

On the contrary, looking at the NATS protocol definition

Open Source. Performant. Simple. Scalable.  
A central nervous system for modern, reliable,  
and scalable cloud and distributed systems.  

It becomes clear that it was built with new modern systems in mind.

One thing to keep in mind is that a
RESTful API could be built on top of NATS across all micro services. The key point is that HTTP is not the best protocol for inter service communication.

Benchmark

In theory it all sounds good, but in practice how much faster is NATS?
To test both approaches a simple benchmark was put together trying to mimic a real world example.

The idea is that a HTTP proxy server will accept client connections and then route the request to the appropriated internal micro service. In one scenario HTTP was used to proxy the messages and on the other NATS.

For the HTTP proxy a node-js express server was used and for the microservice a golang app using the httprouter and the official NATS go-client

To generate the client traffic that would be hitting the proxy the bench-rest tool was used.

Sequential requests with no payload

Overall NATS performed 3x faster than HTTP.
To process 50,000 requests using HTTP as the messaging protocol between the proxy and the micro service it took ~12 mins while using NATS it only took 5.


Sequential requests with 1K payload

When adding some data to the requests the behavior was similar and NATS still processed messages ~3x faster than HTTP.

Parallel requests

When a factor of concurrency was added to the tests that's when NATS performed the best.
To process 50,000 requests with a batch of 2,500 concurrent requests at a time NATS took 1.6 mins while HTTP took 18 mins.

Overall the results shown that NATS beats HTTP both in speed and throughput.

The source code used for the benchmarks can be found here

Bash script notes

A couple of points I wanted to track regarding a bash script a had to write recently:

1 - Do and action if the command execution is successful

Assuming you want to delete a docker container only if it exists. You can first try to inspect that container and if the return value is successful you can then delete it.

if docker inspect $CONTAINER_NAME &> /dev/null;  
  then
  echo "Destroying $CONTAINER_NAME container"
  docker rm -f $CONTAINER_NAME
fi  
2 - Use absolute paths when referring to dependent scripts

A common problem with scripts with multiple dependent files is that a certain path must be hardcoded on the script which can then lead to breaking paths depending on where the user executes that script. To avoid any issues on that regards you can use the bash variable BASH_SOURCE

script_dir=$(dirname "${BASH_SOURCE[0]}")  

So if you need to execute a script relative to the one being executed you can refer to:

$script_dir/my-other-script.sh

Instead of just

./my-other-script.sh

Printing Docker Container IP Address

Just wanted to share a smarcut to print a docker container IP address.

Add the following to your ~/.bashrc or ~/.bash_profile

docker-ip() {  
  docker inspect --format '{{ .NetworkSettings.IPAddress }}' "$@"
}

After sourcing the file you can type on your sell docker-ip myContainerName and get back the IP address associated with the container:

$ docker-ip dnsdock
172.17.0.2  

Got the idea from a post on this stackoverflow thread.

Creating and Deploying a private catalog on the Rancher Platform

With the announcement of private catalog support in the Rancher platform we decided to create a custom catalog for our application stack.

If you are looking for a complete overview and step by step on how to create a private catalog the best place is the January online meetup Building a Docker Application Catalog

In the post I'll guide you through the steps I took to create a catalog for our application, the Aytra Cloud.

The main concepts behind a Catalog on Rancher are

  • Expose docker and rancher compose frameworks to define a catalog blueprint
  • Store templates on git repo
  • Allow configuration and deployment of stacks
  • Support versioning and upgrades

Creating the Catalog file structure

So in a nutshell you will need to have a git repo with the following directory structure

In the config.yml file you can define the basic configuration for you catalog, eg;

name: Aytra  
description: |  
  Aytra Cloud enables our customers to fully realize the potential of cloud computing by allowing them to automatically migrate application workloads across public and private clouds to achieve an ideal balance of security, cost, and performance.
version: 7.5  
category: "Cloud Management Platform"  

To define different versions of the same catalog Rancher uses the concept of numbered directories containing different versions of a docker-compose and rancher-compose templates.

One point to note about the rancher-compose file is that a .catalog section must be defined as well as a questions attribute, for example:

.catalog:
  name: Aytra
  version: 7.5
  description: |
    Aytra Cloud enables our customers to fully realize the potential of cloud computing by allowing them to automatically migrate application workloads across public and private clouds to achieve an ideal balance of security, cost, and performance.
  uuid: aytra-0
  questions:
    - variable: "exo-scale"
      description: "Number of exosphere nodes."
      label: "Number of Exosphere Nodes:"
      required: true
      default: 1
      type: "int"

Adding the Catalog to Rancher

Now with all the catalog file structure defined you can add your private git repo to Rancher under the Admin -> Settings tab.

A couple of tweaks you need to make to enable private git repos:

1. Add the username and password of your git user in the git url: https://username:password@bitbucket.org/org/catalogs.git

2. Bind-mount ssh keys for your git repo in the rancher server container. This approach will work after v056 is released since currently the server container doesn't come with with the openssh binaries installed

Ideally you would be able to configure git users similar to how Jenkins git plugins do it but is not supported at the time.

Viewing the Catalog

The rancher catalog service has a mechanism that polls periodically the git repo for new changes to keep the catalogs up to date. So right after adding the git repo to Rancher you should see your catalog listed under the Applications -> Catalogs menu

Launching the Catalog

Now you should be able to launch your application catalog and get a running stack with your micro services