SmartOS training from Joyent!

It's here! KVM on Illumos

After close to one year of work, Joyent is announcing KVM for SmartOS.

When I started working on the project, I used a skeleton driver I wrote for Illumos, and fleshed out the skeleton by pulling in code from the Linux source. I did not know anything about the Intel virtualization technology, a little about Linux, and had a reasonably thorough knowledge of the illumos kernel and illumos device drivers. I also thought I had a good understanding of Intel architecture and x86 assembler code.

A lot of the work at the beginning involved translation. For instance, Linux uses mutex_spin_lock() to acquire a spin lock. On SmartOS, mutex_enter() is used. Most of the locks on SmartOS are adaptive mutexes. No need for spin locks in most of the KVM code.

For many issues, determining how to port Linux KVM to SmartOS involved understanding what KVM is trying to achieve, how the hardware virtualization works, and how KVM interacts with the rest of the system. At the beginning, I was trying to come up with a rapid prototype that would allow a virtual machine to be created on a single CPU. I tried doing this by pulling in the Linux code that was needed to do this. For areas that I thought could wait, I either did not include that code, or used ifdef’s to comment the code out. This resulted in a driver that was all in a couple of files. The driver source has been re-organized, and now you’ll find most function and data declarations are where you expect.

The Linux kernel code for KVM is about 30,000 lines including comments. This does not include code for other hardware besides x86/x64, and does not include user level. The kernel code on SmartOS is now at 29945 lines, including comments. Over time, pretty much all of the Linux kvm code (for x86) has gotten ported for SmartOS KVM. And it turned out that most of that code is needed, just to get past boot of a virtual machine.

Over time, understanding the capabilities of the Intel technology, and understanding what KVM is doing with the technology are key to getting it to work.

We have released code and an iso, see smartos.org for download links. I hope lots of people try it out and find that it is robust, performs well, and is useful. If you are interested in the implementation, the kvm.c file has an overview comment at the beginning. You may also find the linux documentation useful, and, of course, the Intel documentation, specifically http://www.intel.com/Assets/PDF/manual/253667.pdf and http://www.intel.com/Assets/PDF/manual/253669.pdf.

Some milestones (and approximate dates) in the development:

  • August 30, 2010: Initial “skeleton” Illumos driver placed in git repo. Project is “started”. A few hundred lines of code.
  • November 5: Driver creates and initializes virtual machine and a virtual CPU. Pulling in more and more of the Linux code. At this point, maybe 7,000 lines of code.
  • January 19: Virtual machine launches… and dies shortly afterwards with EPT violation. Well over 12,000 lines of code.
  • February 10: Update user level qemu. Turns out the kernel kvm and user level were not in sync.
  • April 4: Bryan Cantrill and Robert Mustacchi are on the case. Development time shortens.
  • April 25: Boot guest VM, login, run ps and a few other commands all work. When it started working, lots of the code “just worked”. I think there are sections of code (emulation, for instance), that have changed very little from the Linux code.

Between the end of April and now, we have been fixing various bugs, getting rid of a few hundred ifdef’s (i.e., implementing missing code), re-organizing the code from two files containing about 20,000 lines into a file layout more closely in tune with the different components making up KVM, studying performance, and generally testing with different guests.

Share this post:

Vote on HN