virt

Creating API Documentation for Virt Manager

The past few weeks have been nothing but great. The work Kieran and I have been doing has finally start to take some good form and we are coming close to have a beta version release.

We had some bumps across the road but something that really stood out for me was the architectural design of the API.
Most of the times I find myself on the other side of the API, consuming instead of creating.
In this project though, we are having to create the API from scratch.

That’s a very good opportunity to put in practice experience of using other APIs through out the years. More than once I have caught myself complaining that an API is missing this  or that it should have implemented that, etc.
Now I can actually decide what should be included on the API and how the API should behave, which makes me think about all the bad experiences I had with other APIs and make sure I don’t make the same mistakes others did.

With that being said, Kieran and I found ourselves in more than one occasion having to go back and revise some decisions we had made towards the API. Sometimes related to the wording, functionality, response structure, etc.
Spotting errors and fixing them is good, however we also needed to document what the changes were so we could all be on the same page when it came to the latest state of the API. To make sure we could always tell what was the latest features supported by the API without having to go and browser through the source code we decided to create a reference page containing all the API calls and their respective responses.

At first we created the reference page in the README file of the github project, which worked for a bit, but it proved hard to navigate and display the data in a user friendly way, plus we knew that in the future we would need to come up with a better alternative, and that takes me to my next point.

Searching for the right tool

After looking at how other projects were doing in respect to their documentation we started to lay down a plan.

I remembered reading a blog post  a while back listing how the documentation for Popcorn.js was created. The post was from David Seifried, you can check it out here.

Basically the documentation for Popcorn is powered by Jekyll:

Jekyll is a simple, blog aware, static site generator. It takes a template directory (representing the raw form of a website), runs it through Textile or Markdown and Liquid converters, and spits out a complete, static website suitable for serving with Apache or your favorite web server. This is also the engine behind GitHub Pages, which you can use to host your project’s page or blog right here from GitHub.

Jekyll seems like a really good choice since it is already supported by github pages and allow a way to decouple the documentation from the actual project, creating some flexibility when it comes to hosting.

After playing a bit with Jekyll I realized that it could be used for our project, however, I needed to find out a way to display the docs in an organized way and with a nice template, that set me on a different journey looking into several project documentation websites.

The one that really caught my attention was the expressjs.com
The API is listed on a simple and clean way, really easy to navigate.
I decided then to use the same style for the Virt Manage docs.

Having spoken with TJ (the maintainer of express) a couple of times I shoot him an email asking if it was cool to use his template as a baseline, he couldn’t have been nicer and said it was all good.

I then cloned the repo for the expressjs.com on github and started hacking around.

I must say that at first it was a little bit confusing to follow the logic being used to generate the docs, but after changing some things here and there I got a good grip on it.
The structure for the docs is very simple and elegant, focusing on scalability and l8n.

I’ll try to give a short overview of how the docs are structured, which might change in the near future:

Docs root

virt-node-docs-root-ls

The structure is very straight forward.
All the html, javascript and css files are placed in the root.
The directories are:

  • images – All the images used by the docs
  • includes – Code snippets reused across multiple files
  • virt-en – Documentation of all API pieces in English

It is important to note that after defining the skeleton of the docs the only files that need to be changed are the ones in the virt-en directory, which for now are only being listed in English, however, if in the future we decide to add other languages we can easily create another dir, such as virt-fr which can contain all the API documentation in french, without affecting the overall structure of the docs.

Another thing you might also notice is that some files have the .html as well the .jade extension.
The reason for it is that the files are coded using jade, and the Makefile generates the html files which are served to the end user.

Makefile:

[sourcecode language=”bash”]

JADE = ./node_modules/.bin/jade

HTML = index.html
virt-api.html

docs: $(HTML)

%.html: %.jade
$(JADE) –path $< < $< > $@

clean:
rm -f *.html

.PHONY: docs clean
[/sourcecode]

Include dir

virt-node-docs-root-includes

The include directory has some files reused across the docs, such as header, menu and footer for example

API dir

virt-node-docs-api

The API dir is divided into three sections:

  • Server
  • Client
  • Crawler

Each directory contains the documentation for all the API calls for each part of the application

API Server|Client|Crawler dir

virt-node-docs-api-server-ls

The server dir for example has a file for each API call

The file server-listvms-group.jade:

[sourcecode language=”javascript”]
section
h2(id=’server.listvms-group’)
|GET /list/vms

p
|Returns a list with the information of all instances being managed by all libvirt hosts

h3 Format
ul
li
|JSON

h3 Authentication Required
ul
li
|YES

h3 Response Elements
ul
li err
ul
li Can have three different values
ol
li The error message of the command execution
li The stderr output of virsh
li Null if the command is sucessful
li instances (Array of objects)
ul
li id
ul
li Unique Identifier of the virtual machine instance
li name
ul
li The name of the virtual machine instance, which is also a uniquely identifier
li status
ul
li The current status of the instance, possible options:
ul
li Status 1
li Status 2
li Status …
li ip
ul
li Ip of the libvirt host that is managing the virtual machine instance

+js.
{
err: "Error message example"
instances: [
{
id: "Instance ID",
name: "Instance Name",
status: "Instance Status",
ip: "10.0.0.0"
},

{
id: "Instance ID",
name: "Instance Name",
status: "Instance Status",
ip: "10.0.0.0"
},

]
}
[/sourcecode]

In the end that’s what the current API reference page looks like:
Screen Shot 2013-02-18 at 11.29.36 PM

To see the documentation in action you can check out the code on github

Big thanks again to TJ Holowaychuk
You can find his work here and the expressjs documentation here

Creating a Virtual Machine on Linux with KVM, QEMU and Virt

Now a days there are quite a few different options available for virtualization on linux, the most famous ones being:

This blog post will be taking about KVM.

A quick summary of which software will be covered here:

KVM – Kernel Virtual Machine
QEMU – Quick Emulator
Virt – The virtualization API

When KVM and QEMU are used in conjunction, the KVM takes care of virtualizing the CPU and memory management while QEMU emulates all the other hardware resources, such as hard-drives, video, cd-rom, peripherals, etc.

Virt is built on top of libvirt, it provides a set of features to manage virtual machines.

1. Checking for support

Before installing any of the software listed above, you first need to check if your hardware supports virtualization.

[sourcecode language=”bash”]
egrep ‘(vmx|svm)’ –color=always /proc/cpuinfo
[/sourcecode]

That should output a list of flags if virtualization is enabled on your hardware:

[sourcecode]
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constanttsc archperfmon pebs bts repgood aperfmperf pni dtes64 monitor dscpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse41 lahflm dts tprshadow vnmi flexpriority
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant
tsc archperfmon pebs bts repgood aperfmperf pni dtes64 monitor dscpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse41 lahflm dts tprshadow vnmi flexpriority
[/sourcecode]

If the VMX flag is enabled it means your CPU is Intel, SVM means AMD

2. Installing KVM

After checking if the processor supports virtualization, you can start by installing KVM

[sourcecode language=”bash”]
yum install kvm kmod-kvm
[/sourcecode]

There are several version of KVM, here is a list explaining which version is suitable for which need

3. Installing QEMU

QEMU is not available on the default repositories enabled on CentOS, you need to enable the rpmforge-extras repository to have access to the QEMU package with yum.

To enable the repository:

[sourcecode language=”bash”]
wget http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.i686.rpm
rpm -Uhv rpmforge*
[/sourcecode]

Then modify the file:

[sourcecode language=”bash”]
sudo vim /etc/yum.repos.d/rpmforge.repo
[/sourcecode]

Set the enabled key for the [rpmforge-extrs] to 1

[sourcecode language=”bash”]
### Name: RPMforge RPM Repository for RHEL 6 – dag
### URL: http://rpmforge.net/
[rpmforge]
name = RHEL $releasever – RPMforge.net – dag
baseurl = http://apt.sw.be/redhat/el6/en/$basearch/rpmforge
mirrorlist = http://apt.sw.be/redhat/el6/en/mirrors-rpmforge
#mirrorlist = file:///etc/yum.repos.d/mirrors-rpmforge
enabled = 1
protect = 0
gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rpmforge-dag
gpgcheck = 1

[rpmforge-extras]
name = RHEL $releasever – RPMforge.net – extras
baseurl = http://apt.sw.be/redhat/el6/en/$basearch/extras
mirrorlist = http://apt.sw.be/redhat/el6/en/mirrors-rpmforge-extras
#mirrorlist = file:///etc/yum.repos.d/mirrors-rpmforge-extras
enabled = 1
protect = 0
gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rpmforge-dag
gpgcheck = 1
[/sourcecode]

Now you should be able to run:

[sourcecode language=”bash”]
sudo yum install qemu qemu-kvm
[/sourcecode]

And QEMU should be installed

4. Loading the module

With KVM and QEMU installed, it is time to load the kvm module to start playing with the virtualization tools:

[sourcecode language=”bash”]
sudo modprobe kvm-intel
[/sourcecode]

You might get the error:

ERRROR:

FATAL: Error inserting kvmintel (/lib/modules/2.6.32-279.11.1.el6.x8664/kernel/arch/x86/kvm/kvm-intel.ko): Operation not supported

Well, but we checked before and the CPU supports virtualization, right?
Actually most times the BIOS disable virtualization by default, so you need to modify the BIOS settings yourself.
To enable virtualization is very simple, here is a good tutorial explaining the steps.

**Just restarting the computer didn’t work for me.
I had to shutdown my computer and wait a few minutes for the new BIOS settings to take effect.

After enabaling VT for your CPU, you can go ahead and load the modules again:

[sourcecode language=”bash”]
sudo modprobe kvm-intel
[/sourcecode]

To check if they were successfully loaded:

[sourcecode language=”bash”]
lsmod | grep kvm
[/sourcecode]

You should see something like:

[sourcecode]
kvmintel 52890 0
kvm 314739 1 kvm
intel
[/sourcecode]

5. Adding your user to the KVM group

There are two ways to add your user to the KVM group.
The simples and fastest:

[sourcecode language=”bash”]
sudo usermod -G kvm -a diogogmt
[/sourcecode]

Or if you prefer:

Side Note:
To check the script that will automatically load the KVM module every time the computer is booted:
[sourcecode language=”bash”]
cat /etc/sysconfig/modules/kvm.modules
[/sourcecode]

The contents of the file:
[sourcecode language=”bash”]
#!/bin/sh

if [ $(grep -c vmx /proc/cpuinfo) -ne 0 ]; then
modprobe -b kvm-intel >/dev/null 2>&1
fi

if [ $(grep -c svm /proc/cpuinfo) -ne 0 ]; then
modprobe -b kvm-amd >/dev/null 2>&1
fi

modprobe -b vhost_net >/dev/null 2>&1

exit 0
[/sourcecode]

As you can see it checks if the CPU is Intel or AMD and then loads the appropriate module.

6. Installing Virt

The last step is to install Virt, the software that will allow us to manipulate and configure the virtual machine from a nice rich feature API.

[sourcecode language=”bash”]
sudo yum install lib-virt python-virtinst virt-manager virt-viewer
[/sourcecode]

After installing the packages above you could restart the computer so the changes take effect or start the libvirt service yourself:

[sourcecode language=”bash”]
sudo service libvirtd start
[/sourcecode]

7. Running a VM

Now that everything is installed you can test it out by creating a Virtual Machine.

References:
https://fedoraproject.org/wiki/Gettingstartedwithvirtualization?rd=VirtualizationQuickStart
http://linux.die.net/man/8/modprobe
http://www.sysprobs.com/disable-enable-virtualization-technology-bios
https://wiki.ubuntu.com/kvm
http://www.campworld.net/thewiki/pmwiki.php/LinuxServersCentOS/Cent6BaseServer
http://en.wikipedia.org/wiki/QEMU
http://www.linux-kvm.org/page/Guest
Support_Status