Just Enough Developed Infrastructure

Libvirt 0-8-6 and Vmware Esx

Libvirt is a great abstraction library for working in a unified way with different virtualization platforms. It provides a way to interact with Xen, Qemu, KVM, LXC, Openvz, Usermode Linux, Virtualbox and ... you guessed it Vmware ESX.

With the recent version 0.8.6 several things changed for the Vmware Esx level. In this blog post, I'll explore my findings. See also the libvirt esx page

My conclusion is that the support for esx is progressing in libvirt, but quite not there yet. Powershell or the VMware VI/Vsphere Java API is still a better option.

Getting/Compiling libvirt with esx support

Several OS platforms provide libvirt as a package. The propagation of the new version takes some time off course. I made the virsh client work under both Ubuntu and Macosx:

Macosx: libvirt used to be a pain to compile on mac. Thanks to Mitchell Hashimoto author of the excellent tool vagrant I used homebrew and provides the latest version 0.8.3 straight off with esx support.

<% codify(:shell) do %> $ brew install libvirt <% end %>

Ubuntu : I tried Ubuntu 10.10 and it has 0.8.3 when you install the latest package of libvirt, and it doesn't have esx enabled by default: using the rpm <% codify(:shell) do %> $ sudo apt-get install libvirt-bin $ virsh -version 0.8.3 $ virsh -c esx:// error: invalid argument in libvirt was built without the 'esx' driver error: failed to connect to the hypervisor <% end %>

compiling from source

<% codify(:shell) do %> $ wget http://libvirt.org/sources/libvirt-0.8.6.tar.gz $ tar -xzvf libvirt-0.8.6.tar.gz $ cd libvirt-0.8.6

You need a bunch of dependencies before it can properly compile

#checking for XMLRPC... checking libxml2 xml2-config >= 2.6.0 ... configure: error: Could not find libxml2 anywhere (see config.log for details). $ sudo apt-get install libxml2-dev

#configure: error: libnl >= 1.1 is required for macvtap support $ apt-get install libnl-dev

#configure: error: You must install the GnuTLS library in order to compile and run libvirt $ sudo apt-get install gnutls-bin gnutls-dev

#configure: error: You must install device-mapper-devel/libdevmapper >= 1.0.0 to compile libvirt $ sudo apt-get install libdevmapper-dev libdevmapper

checking for LIBCURL... configure: libcurl is required for the ESX driver, disabling it $ sudo apt-get install libcurl4-openssl-dev

$ ./configure --with-esx --prefix=/opt/libvirt $ make $ sudo make install

This will make install virsh in in /opt/libvirt/bin/virsh

<% end %>

Connecting to an esx server

It took me a while to find the correct connect string. Because you are not connecting through libvirtd you have to use double slashes esx:// instead of tripple slashes esx:/// to connect. Also because I was using a test,self signed certificate I needed to instruct virsh to ignore my certificate with the ?no_verify=1 option.

You can also connect over ssh , for more options check the libvirt esx page.

<% codify(:shell) do %>

$ virsh -c esx:/// error: unable to connect to '/usr/local/var/run/libvirt/libvirt-sock', libvirtd may need to be started: No such file or directory error: failed to connect to the hypervisor

$ virsh -c esx:// Enter username for [root]: Enter root's password for error: internal error curl_easy_perform() returned an error: Peer certificate cannot be authenticated with known CA certificates (60) : SSL certificate problem, verify that the CA cert is OK. Details: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed error: failed to connect to the hypervisor

$ virsh -c esx:// Enter username for [root]: Enter root's password for Welcome to virsh, the virtualization interactive terminal.

<% end %>

Looking around on esx server

host information <% codify(:shell) do %>

virsh # nodeinfo CPU model: Intel Core i7 CPU 860 @ 2.80GHz CPU(s): 4 CPU frequency: 2798 MHz CPU socket(s): 4 Core(s) per socket: 1 Thread(s) per core: 1 NUMA cell(s): 1 Memory size: 2096128 kB

virsh # iface-list error: Failed to list active interfaces error: this function is not supported by the connection driver: virConnectNumOfInterfaces

<% end %>

domain information

<% codify(:shell) do %> virsh # list --all

Id Name State

96 puppet-server.example.org running

virsh # dominfo puppet-server.example.org Id: 96 Name: puppet-server.example.org UUID: 564dab50-63a0-8b4f-a1f8-20e4d36efc3b OS Type: hvm State: running CPU(s): 1 Max memory: 786432 kB Used memory: 786432 kB Persistent: yes

virsh # dump puppet-server.example.org puppet.xml error: Failed to core dump domain puppet-server.example.org to puppet.xml error: this function is not supported by the connection driver: virDomainCoreDump

virsh # dumpxml puppet-server.example.org

puppet-server.example.org 564dab50-63a0-8b4f-a1f8-20e4d36efc3b 786432 786432 1 hvm destroy restart destroy

virsh # vncdisplay puppet-server.example.org :1

<% end %>

storage information <% codify(:shell) do %> virsh # pool-list

Name State Autostart

datastore1 active yes

virsh # pool-info datastore1 Name: datastore1 UUID: 5c95caef-e8bd-75da-3d51-8ba2c4849229 State: running Persistent: yes Autostart: yes Capacity: 28.75 GB Allocation: 22.91 GB Available: 5.84 GB

virsh # vol-list datastore1

Name Path

esxconsole-4cd2734f-bd5f-e5f9-9a38-000c29350f5a/esxconsole.vmdk [datastore1] esxconsole-4cd2734f-bd5f-e5f9-9a38-000c29350f5a/esxconsole.vmdk puppet-server.example.org/puppet-server.example.org-000001.vmdk [datastore1] puppet-server.example.org/puppet-server.example.org-000001.vmdk puppet-server.example.org/puppet-server.example.org.vmdk [datastore1] puppet-server.example.org/puppet-server.example.org.vmdk ubuntu-10.10-server-i386.iso [datastore1] ubuntu-10.10-server-i386.iso

virsh # vol-dumpxml "[datastore1] puppet-server.example.org/puppet-server.example.org-000001.vmdk"

puppet-server.example.org/puppet-server.example.org-000001.vmdk 6000c29d-0c39-c276-a97a-5af3d9b806c9 10240000000 1997537280 [datastore1] puppet-server.example.org/puppet-server.example.org-000001.vmdk 00 0 0 <% end %>

Creating/Removing a new esx volume

This is what a new volume xml file looks like. Note that the capacity and allocation had to be the same: otherwise I would get an error <% codify(:shell) do %> error: internal error Unsupported capacity-to-allocation relation <% end %>

So I found no way to create incremental vmdk files.

<% codify(:xml) do %>

test.example.org/test.example.org.vmdk 6000c29d-0c39-c276-a97a-5af3d9b806c1 10240000000 10240000000 [datastore1] test.example.org/test.example.org.vmdk 00 0 0 <% end %>

<% codify(:shell) do %> virsh # vol-create --file disk.xml --pool datastore1 Vol test.example.org/test.example.org.vmdk created from disk.xml <% end %>

Removing the volume didn't work though: <% codify(:shell) do %> virsh # vol-delete "test.example.org/test.example.org.vmdk" error: Failed to delete vol [datastore1] test.example.org/test.example.org.vmdk error: this function is not supported by the connection driver: virStorageVolDelete

virsh # vol-wipe "test.example.org/test.example.org.vmdk" error: Failed to wipe vol [datastore1] test.example.org/test.example.org.vmdk error: this function is not supported by the connection driver: virStorageVolWipe <% end %>

Creating/Removing a new esx domain

I adapted this xml file from the existing domain. Note that the memory needs to be multiple of 4096.

<% codify(:xml) do %>

test.example.org 262144 1 hvm destroy restart destroy

<% end %>

I first tried to create the new domain:

<% codify(:shell) do %> virsh # create test-domain.xml error: Failed to create domain from new.xml error: this function is not supported by the connection driver: virDomainCreateXML <% end %>

I found a blogpost that explains there is a difference between create and define: Create creates a transient domain and starts it ESX(i) doesn't have this transient semantic for it's domains,therefore libvirt can't support virDomainCreateXML for it.

<% codify(:shell) do %> virsh # define test-domain.xml Domain test.example.org defined from test-domain.xml <% end %>

<% codify(:shell) do %> virsh # start test.example.org Domain test.example.org started

#Snapshot it virsh # snapshot-create test.example.org Domain snapshot 1291744079 created

#List the snapshots virsh # snapshot-list test.example.org

Name Creation Time State

1291744079 2010-11-28 22:46:24 +0100 running

#Now we stop it virsh # destroy test.example.org Domain test.example.org destroyed

#Now remove the definition virsh # undefine test.example.org Domain test.example.org undefined

<% end %>

Tips and gotchas

enable debugging to see what's happening with setting the LIBVIRT_DEBUG flag <% codify(:shell) do %> $ export LIBVIRT_DEBUG=debug <% end %>

or starting virsh in debug mode

<% codify(:shell) do %> $ virsh -d 5 <% end %>

On my mac I had to set the TMP_DIR variable to /tmp otherwise I would get an error : <% codify(:shell) do %> error: /var/folders/o2/o205qRiGGyiU67-ImZI4iU+++TI/-Tmp-//virshY3WtwS.xml: temporary filename contains shell meta or other unacceptable characters (is $TMPDIR wrong?) <% end %>