Vagrant NFS Retryable exception

Last week I hit an odd issue with Vagrant. My environment was working fine when suddenly while resuming my box I started to get the following exception printed in the console:

INFO retryable: Retryable exception raised: #<Vagrant::Errors::LinuxNFSMountFailed: The following SSH command responded with a non-zero exit status.  
    Vagrant assumes that this means the command failed!

After checking VirtualBox I could see my virtual machine was running and I was able to open a shell connection to it, however, the NFS shares were not mounted.

That lead me to a path of trying to figure it out what could be the cause of the problem. In the end I didn't find a clear answer, the closest I got was this answer from Hashimoto: I understand that this is an intermittent issue for people. Iv'e even seen it myself. I'm still not sure what exactly causes this or how to get a solid reproduction. Or, how to fix it. Because of that, I'm going to close it since it is a rare issue. If someone can shed more light, I'd be happy to fix. Thanks.

What I did manage to do was gather a few steps that helped some people solved the Vagrant problems with NFS, they are listed below:

Before trying any of the options below make sure to export the VAGRANT_LOG=DEBUG environment variable to get more details in the console:


1 - Configure firewall

On the mac, under System Preferences > Security & Privacy - Firewall
Make sure the Automatically allow signed software to receive incomming connections is checked

A more detailed firewall rule list can be found on this github issue

2 - Install vagrant VBGuest plugin

To install the plugin:

vagrant plugin install vagrant-vbguest  

The plugin's github page

3 - Restart vboxnet0 interface

Bring the interface down and up

$ sudo ifconfig vboxnet0 down
$ sudo ifconfig vboxnet0 up

4 - Restart NFSD daemon

First remove all the entries for your vagrant box in the /etc/exports file and then restart the nfsd daemon

sudo nfsd restart  

5 - Install NFS dependencies on the client

In the case of an Ubuntu virtual machine make sure the nfs-common and nfs-kernel-server are installed

apt-get install nfs-common nfs-kernel-server  


Resizing Kolla Vagrant CentOS base box

At the time of this writing Kolla uses the puppetlabs/centos-7.0-64-puppet as the base box for the Vagrant Virtual Box provider.
The main issue with the selected base box is that it only has a 20GB root disk. However, Kolla needs at least a 40GB disk when doing a All In One installation.

I ran into the following error when using the default 20GB base box:

The following SSH command responded with a non-zero exit status.  
Vagrant assumes that this means the command failed!  
cat /tmp/vagrant-network-entry_enp0s8 >> /etc/sysconfig/network-scripts/ifcfg-enp0s8  
Stdout from the command:  
Stderr from the command:  
cat: write error: No space left on device  

The temporary solution just to get Kolla installed using Vagrant without any further modifications is to resize the puppetlabs/centos-7.0-64-puppet base box to have a larger disk size.

Create Vagrant VM

After cloning the kolla repository, navigate to kolla/dev/vagrant and run vagrant up
Then shutdown the VM vagrant halt

Find Disk to be resized

List all VMS VBoxManage list vms

Keep note of the VM name
Example: "vagrant_operator_1446740538626_12437" {29db98c9-b085-4ecc-be9b-dba01d8c2256}

Find the VM disk: VBoxManage showvminfo vagrant_operator_1446740538626_12437 | | grep vmdk
Example: IDE Controller (0, 0): /Volumes/diogogmtdata/VirtualBox VMs/vagrant_operator_1446740538626_12437/packer-centos-7.0-x86_64-virtualbox-vagrant-puppet-1438003672-disk1.vmdk (UUID: 3ca3b592-784e-4dc2-9e84-af0b63bec83f)

Resize the disk

Convert VMDK disk to VDI: VBoxManage clonehd packer-centos-7.0-x86_64-virtualbox-vagrant-puppet-1438003672-disk1.vmdk vagrant-puppet-disk1.vdi --format vdi

Resize the VDI disk: VBoxManage modifyhd vagrant-puppet-disk1.vdi --resize 61440

Attach resized disk into the VM

Find the storage controller name: VBoxManage showvminfo vagrant_operator_1446740538626_12437 | grep Storage
Example: Storage Controller Name (0): IDE Controller

Attach disk into the VM: VBoxManage storageattach vagrant_operator_1446740538626_12437 --storagectl "IDE Controller" --port 0 --device 0 --type hdd --medium vagrant-puppet-disk1.vdi

Note: By attaching as device 0 automatically will detach the existing disk

Create new primary partition

Start again the VM: vagrant reload && vagrant ssh

Using fdisk: fdisk /dev/sda

Command: n

  • Partition type: p (primary)
  • Partition number (3,4, default 3): 3
  • Use default value for first and last sectors

Command: t

  • Partition number (1-3, default 3): 3
  • Hex code (type L to list all codes): 8e

Command: w

Reboot VM: reboot

Extend partition

Create physical volume: pvcreate /dev/sda3

Extend volume group: vgextend centos /dev/sda3

Extend partition: lvextend -l +100%FREE /dev/mapper/centos-root

Since CentOS uses XFS as the file system you can't use resize2fs to resize the partition.

Instead use xfxgrowfs:
Install xfx
growfs: yum install -y xfsprogs.x86_64
Resize partition: xfs_growfs /dev/mapper/centos-root


Executing script on the host when running Vagrant commands

When running Vagrant commands there might be the need to execute something on the host as a pre setup requirement.

Since Vagrant is built with ruby it is possible to execute a shell script utilizing the system utility


In some cases you might want to run a script only in a certain command of Vagrant, for example, when reloading the Vagrant box you might need to check if your host has all gateway routes configured or any other pre setup task.

  if [ #{ARGV[0]} = 'reload' ]; then
    echo 'Setting up routes on the host when reloading Vagrant box'