Desktop VMs

A free desktop VM supervisor (Hyper-V) is built into Windows 10. It is based on the industrial strength Windows Server and Azure versions of Hyper-V used by Microsoft to support Datacenter and Cloud based virtualization. The desktop version inherits a high performance, reliable platform originally designed for use by professionals. As a result it may more attractive for IT staff but more complicated for end users.

When Hyper-V is enabled, the computer starts up differently. Hyper-V loads first and controls memory management and some devices. When Windows 10 loads, it discovers it is running under Hyper-V and loads some different device drivers and system components than it would if it was running on its own. Hyper-V was originally designed for Windows Servers that run in a machine room unattended and depend mostly on their LAN adapters and disk controllers. When they moved it to desktop systems starting with Windows 8, they encountered new third party device drivers that you don't find on a server (audio adapters and video cards designed for gaming) that initially did not work as well under Hyper-V. However, Microsoft has worked with vendors to fix those problems, and today a Windows 10 desktop works as well with Hyper-V as without it. This has become important as new Windows security features are making Hyper-V an effective requirement for developers and creators, although I still make no claim about gaming. 

The other two options are Oracle VirtualBox and VMWare Player. These programs run as ordinary applications on your Windows 10 system, although when they are running they demand exclusive use of the virtualization support hardware in the CPU chip. You can run as many VMs under either system as you can fit into your memory, but if you are running a VirtualBox VM you cannot start Player, run Player and you cannot start VirtualBox, and because Hyper-V starts before anything else and runs all the time, it prevents the use of both the other systems.

Player is the limited function free version of a $200 VMWare product called Workstation. It has been stripped of many features that are only available in the product. VirtualBox is fully featured, but it is always in development and the newest changes seem to be in perpetual Beta, or are "experimental" and mostly don't work. Both are fine for doing the core business of running a VM, so the difference is that everything in Player has been tested and probably works, but what you want may not be there. More features are in VirtualBox, but some "nice to have" things don't work.

More specifically:

• VirtualBox and Player emulate a complete desktop computer, simulating standard video cards and LAN adapters. They can therefore run any system that ran on some pretty old hardware. Hyper-V has some emulation of "legacy devices", but it mostly expects that the operating system in the VM will have drivers for some special Hyper-V only devices for video and LAN. Since all modern versions of Windows and Linux have the Hyper-V support, this only limits you if you want to boot up something like the OS/2 operating system from 1994.
• VirtualBox and Player simulate a keyboard and mouse attached to the VM, and display on your real monitor a window whose contents are the image of the virtual screen output from the simulated video card. However, since Hyper-V was initially developed for servers, and servers typically don't have a keyboard, mouse, screen, or a meaningful video card, Microsoft instead adopted the strategy of "connecting" to the VM using the same remote client software (Remote Desktop Client for Windows systems and SSH, Xwindows, or something like RealVNC for Linux). Actually none of the three options provides decent support for running a modern video game inside a VM.
• VirtualBox and Player are ordinary applications that have to use the Windows host through the standard services available to Word and your Web Browser. Hyper-V support for virtual disks (in VHD format) or virtual network connections to the VMs is built into the Windows kernel. This was necessary to get the best performance on big servers with lots of VMs, but the performance advantage (at a smaller scale) becomes available to even desktop users.
• Until recently, Microsoft did not really care about Linux. Ubuntu and Red Had put Hyper-V drivers into their Linux distributions because they wanted to run on the Microsoft systems, but Microsoft did nothing to Hyper-V to make it friendly to other systems. This has begun to change recently, but you will find more Linux friendly support on VirtualBox or Player.

Companies do not make money giving away free software. So the future of these three products is partially determined by how development is funded.

• VirtualBox was created by a company that was purchased by Sun Microsystems. Sun Microsystems was committed to the philosophy of "cross-platform", so VirtualBox is the only option that runs on Windows, Mac, and Linux. Then Oracle bought Sun, and accidentally inherited this minor open source software development project. It was useful enough to keep going, not expensive to support, but it was never going to compete with Microsoft and VMWare for corporate Datacenter infrastructure, nor would it compete with Azure, AWS, or Google for Cloud business. It is not clear how Oracle expects to make any money with it, and Oracle has recently become less generous supporting projects it cannot turn into revenue. They keep control of a necessary Extensions package, so it isn't completely open source.
• VMWare is in the business of building Datacenter VM solutions. Player is a function-limited sample of their real stuff. For serious development on the desktop you need at least a $200 version of Workstation, and they really want to sell you large scale machine room servers. VMWare generates new releases of Player infrequently, and they actually test it, so it is more likely to do all the things it claims to do, while VirtualBox releases whatever it has every month or so and the newest features are in beta and probably don't work.
• Microsoft developed Hyper-V for its Windows Server and Azure business. Because Windows 10 and Windows Server share the same kernel, most of the OS, and standard drivers, it was easy to enable Hyper-V in the Windows desktop environment. However, the first release in Windows 8 had problems. Servers often have trivial local video support, RAID disk controller cards, and powerful LAN adapters, but they don't have the odd end user adapter cards you buy for $10 on Amazon. It had been easier to verify correct operation on servers with restricted hardware options. Windows 10 needed to tighten up its standards for testing and certifying device drivers, and the new standards ensured that new mainstream hardware supported Hyper-V. So today running Hyper-V on a desktop is more successful. The benefit is an industrial strength virtualization supervisor with all the features of the more sophisticated products available free if you just turn it on. Recently Microsoft has added sequence of security oriented Windows features that depend on Hyper-V to provide protected "Sandbox" environments to isolate questionable Web services or desktop applications. Although Hyper-V remains an option for now, it may eventually become a necessity to use all these neat new features.

In Windows 8 where Hyper-V did not work as well and could cause problems with the system once enabled, people chose Player as a safe option for light weight use. Today, people with a Mac choose VirtualBox because it works across platforms. However, if you run Windows 10 on your desktop and mostly want to run Windows 10 or Windows Server virtual machines, there is no reason not to simply enable Hyper-V.

Simulate Hardware or Provide Services

The first generation of any Virtual Machine supervisor will support the emulation of existing real hardware devices. Typically they choose to emulate standard disks, simple video cards, and some relatively old LAN adapter that all systems support.

However, everyone quickly re-learns a lesson IBM discovered in the 1960's, that emulating the tiny intricacies of real hardware is inefficient. It is better to offer the option of device drivers for synthetic devices that do the same thing as real hardware, but which exist only Virtual computers and are easier to emulate. Eventually you find changes in the system kernel that do away with simulated devices at all and communicate directly with the VM supervisor.

One of the most important optimizations is a "device" nobody things about: the timer chip. Timers as they are implemented in PC hardware generate a lot of interrupts and are quite inefficient. Simulating the interrupts becomes much worse in a VM. However, the host system has a full set of timer services, and it can certainly tell the VM "what time is it" if you change the VM kernel to as the host instead of using the simulated hardware.

This approach has a fancy name called paravirtualization. It replaces hardware simulation with an arrangement where the VM asks the host operating system to perform many of the same services the host is already performing for its applications. The operating system inside the VM has to be modified to detect whether it is running on real hardware or inside a virtual machine, and what VM supervisor is present. If it has support for that VM supervisor, then it replaces hardware functions with service calls.

Integrate the Desktop

If you have a folder open on the host desktop, and there is a corresponding folder open in the window of a Windows system running in a VM, it seems natural to want to drag and drop a file from one window to the other. Similarly, if you have editors open on both systems, it seems natural to select some text and do a Copy in one editor and then select a destination and to a Paste in the other editor. This is not about hardware or drivers. It is about connecting the two operating systems and integrating their "Clipboard" functions.

Microsoft recently added a network shared clipboard to Windows 10. Some third party packages do better, and provide some shared clipboard services that work between Windows and Linux. VirtualBox and Player have some support for cut-and-paste or drag-and-drop between hosts and virtual machines. It is not seamless, sometimes not reliable, and should be regarded as an aspiration or a work in progress rather than as an expected feature.

However, while Player and VirtualBox generate a Windows application that displays the simulated content of some virtual video adapter in an application window, Hyper-V creates an instance of the standard Windows Remote Desktop Client to connect to the VM. Running as an local Windows application, it can cut and paste text between systems, and while you may not be able to drag and drop files, Remote Desktop Protocol provides a way to share disks from the Client (the host) to the Server (the VM).

When the VM is running Linux, you may prefer running it without a simulated monitor, then connect to it over the network with any Linux remote client protocol like SSH, Xwindows, or RealVNC.

Virtual Devices

Disk

Virtual Machines can use physical disks, but most of the time they use "disk image" files that simulate a hard disk. The disk image contains records representing disk sectors, including partition tables. Each VM supervisor seems to have created its own format (VDI for VirtualBox, VMDK for VMWare, VHD for Hyper-V). VirtualBox supports all three formats and has a utility to convert from one to the other. Disk image files are typically dynamic, so they start small and grow as data is written to previously unused sector numbers.

There is an exposure if the system crashes while a dynamic disk is being expanded. If it would be difficult to recover, you might create the disk image file with a fixed size so it doesn't example. VirtualBox claims it can recover from crashes better if the disk is a VDI format, and Hyper-V has an improved version of VHD called VHDX. The VHD format is supported by both systems, so a strategy is to use dynamic VHD files for distribution to other users (and zip them up to save space when they are archived or shared), but then when you get one of these files and intend to use it personally you are free to convert it to one of the other formats.

Windows can "mount" a VHD or VHDX file as if it were a portable disk (like a USB drive). The Windows Disk Manager has an Attach VHD function that lets you mount it and assign letters to the volumes.

The Windows Boot Configuration (BCD) file also allows you to add a VHD or VHDX to your boot menu. If you select that option, the system boots from the disk image file as if it were a regular hard disk.

Hard disk image files simulate a bare hard drive, so the first step must be to create one of the two formats of partition tables at the start of the disk. For real disks and modern machines there is no reason to use the legacy Master Boot Record (MBT) format created for the first IBM PC. Modern computers support the newer and better Guid Partion Table (GPT). However, Virtual Machine supervisors often emulate really, really old hardware that provides a lowest common denominator supported by all sorts of operating systems. VirtualBox has what it has called "exprimental" support for the EFI boot mode needed to boot off a GPT disk, but they don't recommend you use it. So to make system disks portable, you generally have to use the MBR format. If you are only going to use Hyper-V, then you can create a GPT system disk and use it with a "Generation 2" type of Virtual Machine, but then you cannot share that disk with others who use VirtualBox because they run on Macs. For data disks that you do not boot, GPT is the obvious choice.

Every Hard Drive is assigned a unique identifier when it is first attached to any operating system. If you copy a disk image file (copy a.vhd b.vhd) then you copy the unique identifier. Windows and Linux will not mount two disks with the same unique identifier at the same time. VirtualBox, in fact, doesn't want two different disk image files on the same host computer, even if they are attached to two different virtual machines. So if you want a copy of the same data or the same system, it is better to use the VirtualBox "clone disk" utility, which changes unique identifiers, rather than just making an ordinary copy of the file. An ordinary copy can be used to make a backup, or to share a copy of the disk image to someone who will put in on another host computer.

One unresolved problem is how to organize your disk image files. One idea is to create a directory or directory tree that is a library of different disk images. This makes it simpler to classify operating system disks with special features, or data disks with special content, but it obscures the relationship of disk images to particular VMs. The other approach is to embed disk image files in the directory representing a VM to which they are attached. This is really just a software version of a problem we have with physical disks. Some disks we put inside the computer case, and while you can take the computer apart and remove them, they are normally bound to the particular system into which they are installed. Other disks, particularly small 2.5" SATA and SSD devices, can be attached to a USB 3.1 cable and dynamically connected to any computer as needed. So whether you have physical disks or disk image files, you have to decide which are associated with particular machines (and store them inside the case or inside the directory of the VM), and which are portable across machines (and then it makes sense to store them in a library classified by use or content).  Of course, library disk images can only be used by one system at a time.

Windows has the ability to mange "differencing" disk image files. VirtualBox has a similar idea called a "Multi-Attach" disk that is a different way of looking at the same idea.

To create a differencing disk, you need a PowerShell command that is installed when you enable the Hyper-V Windows optional feature:

New-VHD -ParentPath c:\Server2016.vhd -Path c:\TestAD.vhd -Differencing

This creates a new VHD file named TestAD.vhd that is very small, yet it appears to contains everything that was in the input Server2016.vhd disk image. That is because the two files are now internally linked. When you open the TestAD.vhd file, the Windows kernel automatically opens the Parent Server2016.vhd file as well. If you write any changes to sectors in the TestAD disk, they stay in the TestAD.vhd file and are subsequently read back from that file. If you ask for a sector that is not in the TestAD.vhd file, then Windows reads it from the parent Server2106.vhd file. You have to treat the Parent disk as Read-Only. If you change it, that invalidates all the differencing disks based on it. Each differencing disk contains its own set of changes against the shared Parent.

VirtualBox has essentially the same idea with a Multi-Attach disk, but you cannot create standalone differencing disk files. You simply mark C:\Server2016.vhd as "Multi-Attach", and then when you create a new VM and attach that disk to it, under the covers VirtualBox creates a new anonymously-named differencing disk image file inside the virtual machine directory. You cannot do more than that unless, as Microsoft has done with Windows 10, you can add support for differencing disks to the kernel of the real operating system.

Because Parent disk are Read-Only and cannot be changed without invalidating all the differencing disks based on them, you either never put maintenance on any programs they contain or you have to put maintenance on every differencing disk or VM based on them. Therefore, when you use this trick try to keep all your data files and unrelated programs on a separate disk that you mount to the system. When you eventually do want to change the system, you may have to discard all the differencing disks, which means you will lose any changes and customizations you made. Or you can combine the Parent content with the changes you made to particular differencing disk to create a new Parent, but now you need room on your real hard disk for multiple copies of the whole disk image.

Network

Hyper-V can simulate a "legacy" hardware LAN adapter, but it is more efficient with a synthetic Hyper-V LAN adapter whose drivers are built into Windows and most Linux distributions.

The option is how to configure the simulated network to which the VM appears to be connected. The options are influenced by the basic difference between Hyper-V and VirtualBox:

• VirtualBox runs as an application inside the host operating system. A service programmed inside the VirtualBox application connects the VM to the real network through either a simulated bridge or routing function. You have to create a special type of network to allow the host to connect to the VM.
• Hyper-V creates VMs that run parallel and external to the host Windows system. The host and VMs are connected by a synthetic data bus which makes it natural to structure any network as a LAN connecting the VMs to the host and treating the host as a router that connects the VMs to the real network. Unlike VirtualBox, any network that allows the VMs to access the internet also allows the host to talk to the VMs.

VMs in Hyper-V are simpler because you can do everything you want with the default networks, but it also adds more synthetic LAN adapters to the host Network Connections configuration. VirtualBox only generates synthetic Network Connects when you ask for them, but they are specialized and a given VM may need more than one LAN to do everything you want.

Bridge

Every Ethernet adapter has a hardware address and gets assigned an IP address. When you choose a Bridge connection in VirtualBox or Hyper-V, the host Windows system and the VM share the Ethernet adapter. The adapter is assigned a second Ethernet hardware address and a second IP address. Now both the host and the VM appear to be machines on your local network.

Simple NAT

Network Address Translation is the technique used by your home router to connect all the computers in your house to the Comcast ISP using only the one IP address Comcast has assigned to your home router. 

In VirtualBox, when you configure a LAN adapter you can select NAT as the network management option from a pulldown menu. Since VirtualBox is simply an application in the host Windows system, the NAT function is simply part of the LAN adapter simulation logic in the application. When the VM sends IP traffic over the adapter, the VirtualBox LAN sumulation logic changes the IP address and port number and then sends the data using the Windows network API. VirtualBox certainly sees all the data, but it doesn't expose the VM to anything else running on the host OS.

In Hype-V the VM runs separate from the host OS. The host and VM communicate over a LAN simulation provided by the hypervisor. Since the VM is external rather than hiding inside an application, the simulated network between the host system and the VM can also be used by other programs. So the host and VM can share files or make database requests across systems. NAT is the Default network connection created by Hyper-V when you install it, so it requires no configuration. It does, however, generate one emulated Network adapter added to the Network Connections configuration on the host computer. 

Host Only Network

This is the name that VirtualBox uses to reference a simulated LAN adapter created in the host operating system that is then virtually connected to one or more VMs. Unfortunately, in VirtualBox this simulated network doesn't automatically get a NAT function added, so if this is the only type of LAN adapter defined to a VirtualBox VM, then the VM can talk to the host computer but not to the Internet. Of course, there are several separate software solutions that add NAT services to a Windows system (Internet Connection Sharing, NAT32, and WinGate all come to mind). Unfortunately, although VirtualBox has NAT services built in, a pure NAT adapter doesn't allow the host to talk to the VM, while a Host-Only network doesn't allow the VM to talk to the Internet. As a results, in VirtualBox you have to define two LAN adapters (one NAT and one Host-Only) to get both functions, while in Hyper-V the Default combines both services and requires no extra configuration.