Installing OpenStack, Quantum problems

During the following weeks we plan to expand more on the subject of setting up an OpenStack cloud using Quantum.
For now we have been experimenting with different Quantum functionality and settings.
At first Quantum might look like a black box, not due to its complexity, but because it deals with several different plugins and protocols that if a person is not very familiar with them it becomes hard to understand why Quantum is there in the first place.

In a nutshell Quantum has the role to provide an interface to configure the network of multiple VMs in a cluster.

In the last few years the lines between a system, network and virtualization admin have become really blury.
The classical unix admin is pretty much non existent now a days since most services are offered in the cloud in virtualized environments.
And since everything seems to be migrating over to the cloud some network principles that were applied into physical networks in the past some times don’t translate very well to virtualized networks.

Later we’ll have some posts explaining what technologies and techniques underlie the network configuration of a cloud, in our case focusing specifically on OpenStack and Quantum.

With that being said, below are a few errors that came up during the configuration of Quantum:

1. ERROR [quantum.agent.dhcp_agent] Unable to sync network state.

This is error is most likely caused due a misconfiguration of the rabbitmq server.
A few ways to debug the issue is to:
Check if the file /etc/quantum/quantum.conf in the controller node(where the quantum server is installed) has the proper rabbit credentials

By default rabbitmq runs on port 5672, so run:

netstat -an | grep 5672

and check if the rabbitmq server is up an running

On the network node(where the quantum agents are installed) also check if the /etc/quantum/quantum.conf have the proper rabbit credentials:

If you are running a multihost setup make sure the rabbit_host var points to the ip where the rabbit server is located.

Just to be safe check if you have a connection on the management networking by pinging all the hosts in the cluster and restart both the quantum and rabbitmq server as well the quantum agents.

2. ERROR [quantum.agent.l3agent] Error running l3nat daemon_loop

This error requires a very simple fix, however, it was very difficult to find information about the problem online.
Luckily, I found one thread on the mailing list of the fedora project explaining in more details the problem.

This is error is due to the fact that keystone authentication is not working.
A quick explanation – the l3 agent makes use of the quantum http client to interface with the quantum service.
This requires keystone authentication. If this fails then the l3 agent will not be able to communicate with the service.

To debug this problem check if the quantum server is up and running.
By default the server runs on port 9696

[email protected]:/home/senecacd# netstat -an | grep 9696
tcp 0 0* LISTEN

If nothing shows up is because the quantum server is down, try restarting the service to see if the problems goes away:

quantum-server restart

You can also try to ping the quantum server from the network node(in a multihost scenario):

[email protected]:/home/senecacd# nmap -p 9696

Starting Nmap 5.21 ( http://nmap.org ) at 2013-01-28 08:07 PST
Nmap scan report for folsom-controller (
Host is up (0.00038s latency).
9696/tcp open unknown
MAC Address: 00:0C:29:0C:F0:8C (VMware)

Nmap done: 1 IP address (1 host up) scanned in 0.04 seconds

3.ERROR [quantum.agent.l3agent] Error running l3nat daemon_loop – rootwrap error

I didn’t come across this bug, but I found a few people running into this issue.
Kieran already wrote a good blog post explaining the problem and how to fix it

You can check the bug discussion here

4. Bad floating ip request: Cannot create floating IP and bind it to Port , since that port is owned by a different tenant.

This is just a problem of mixed credentials.
Kieran documented the solution for the issue here

There is also a post on the OpenStack wiki talking about the problem.


This should help fixing the problems that might arise with a Quantum installation.
If anybody knows about any other issues with Quantum or has any suggestions about the problems listed above please let us know!

Also check the official guide for other common errors and fixes

VMware Fusion MacBook Pro 2012 3D Acceleration Bug

After a couple of hours trying to figure it out why every time I tried to log in to my Ubuntu VM using VMware Fusion on a mac the vm would crash and restart itself, I came up with a thread in the VMware Communities forum explaining the reason + possible solution for the problem.

Basically what happened was that the Ubuntu would start just as expected, but after logging in it would popup this message:
Screen Shot 2013-01-20 at 9.31.51 PM

Quoting their explanation:

…If you are running a mid-2012 MacBook Pro with the latest Apple updates, you’ll likely encounter crashes unless you disable 3D graphics: With your VM powered off, go into Virtual Machine > Settings, then choose Display, and turn off the Accelerate 3D Graphics option.
For others encountering the same problem: if you only run Windows VMs, you would also have the option of reverting to Fusion 5.0.1 to avoid this issue.

Indeed, after disabling 3D acceleration support everything came back to normal.

Firefox Bug 714071

Working on Bug 714071 introduced me to another layer of Firefox.
So far all I’ve been doing was working with c++ code, specifically related to the MouseLock API.
Bug 714071 on the other hand was focused on the js layer.

A brief summary of the bug:

Fix a problem with the Statistics video control

When the showing statistics option of a video was on, and the fullscreen was toggled the video would stop displaying the statistics but the menu would not be updated.
The goal for the bug was to keep showing the statistics when toggling between fullscreen.

The code that displays the statistics on a video is the following:

 showStatistics : function(shouldShow) {  
 if (this.statsInterval) {  
 this.statsInterval = null;  

 if (shouldShow) {  
 this.video.mozMediaStatisticsShowing = true;

 this.statsOverlay.hidden = false;  
 this.statsInterval = setInterval(this.updateStats.bind(this), this.STATS_INTERVAL_MS);  
 } else {  
 delete this.video.mozMediaStatisticsShowing;  
 this.statsOverlay.hidden = true;  

Everytime the video was loaded on the page, or toggled in fullscreen, the init method would be called and an event listener would be attached to listen for the “Show Statistics” option click. However, the init method would initialize the video with a fresh config, so if the statistics were being displayed it would be hidden after switching between fullscreen mode.


The solution was to add a check on the setUpInitialState method to activate the statistics on the video if they were being displayed before toggling the fullscreen.

 if (this.video.mozMediaStatisticsShowing) {  

That apparently solved the problem and the statistics are preserved even when toggling fullscreen.

Working on this bug made me very curious. How could some javascript code interact with c++ at run time.
I knew about the XPCOM object model used in Firefox and lately I’ve started to read more about XUL and the Gecko engine . This was the perfect time to start digging more deep and learn more about the Firefox foundations.
Using the videocontrols as a starting point I went to mxr and started searching some code.
From my initial search I think I found where the controls were loaded in c++. Now I just need to figure it out how all that happens :)

videoscontrols.xml – define the video controls

nsCOMPtr mVideoControls;
nsVideoFrame::CreateAnonymousContent – where the menu gets attached to the video element

nsNodeInfoManager – apparently used to load the video controls as well the poster image
nsNodeInfoMangager::GetNodeInfo – looks like the loading happens here

Manage XPCOM objects?

NSTrustedNewXULElement – where the XUL element gets created and casted to an nsIContent
TrustedNewXULElement declaration
NS_TrustedNewXULElement definition

Keep digging and find how the pieces fit together!