Ken's Virtual Reality

Jason Boche posted an interesting article on his blog today, and I thought I'd offer my thoughts on it.

First, here's the article from Virtualization Review magazine that started all this furor!

My guess is that part of the difference comes from both Hyper-V & Xen requiring VT capable CPUs (i.e. the VM always runs within a VT jail) while ESX supports binary translation (BT) for some 32-bit x86 instructions. The first generation of chips that supported VT weren't very good, and VMware's BT would often do a better job of executing the protected instructions than the hardware assist provided by the CPU. Intel has gotten better in the hardware support for virtualization which is shown in the subject test.

I've taken the data that was provide and normalized it and dropped it into some charts to help visualize the differences in performance between the three platforms. For each chart, the data was broken out by test and then normalized to percentages.

In the first two charts (CPU Performance & RAM Performance), longer bars represent better performance, while in the third & fourth charts (Disk Performance & SQL Performance), shorter bars are better.

When looking at CPU performance, ESX takes a beating in Test 1 & Test 2; however, in Test 3, it fares much better.





I supsect that CPU performance and RAM performance are inter-related - better CPU performance will yield better RAM performance. This suspicion is reasonably well borne out by the test results.





In disk performance, I don't know enough about how Hyper-V & Xen manage disk I/O to make much speculation here. Do they do caching above and beyond what is provided by the RAID controller? ESX does not. Were the .vmdk files (and the .vdf files) properly aligned when they were created? Did each hypervisor use the same block size? I don't know. I also have no idea why disk metrics were excluded from the Test 1 results - a question for the original authors, I suppose.

Remember, shorter bars = better performance in this graph.




For SQL performance, I took the elapsed time, converted it to seconds, and then into relative percentages. Here again, shorter bars represent better performance. Comparing the Test 2 & Test 3 results for Disk and SQL performance, it's clear that there's a close correlation between the two. Based on this, I surmise that you could extrapolate Test 1 disk performance from the Test 1 SQL performance numbers.





I do agree with Jason's remarks in his comment above - this was an "apples to apples" comparison, at least as far as we can tell from the information provided in the source article. It may have to do with the fact that both Hyper-V and Xen have both just completed a major rev and ESX is on the cusp of doing the same (I'm interested to see this same test with the next version of ESX, when it's available) - or it could simply be that VMware's got some performance tuning to do. Either way, an interesting article worth discussion.

Regardless of the differences shown in the tests that were performed, I feel that this particular test is relevant only for the SMB space. The enterprise environment isn't going to care so much about the performance differences shown here. The true differentiators in the enterprise are the very management features that were excluded from this test.