Let’s say I have my own hosting platform based on vSphere, where I run most of my customers, as a service provider, on a decent cluster with multiple nodes and shared storage and all that jazz.
Let’s further say that I have customers that have local Hyper-V environments running on single boxes here and there, because those customers have chosen, because Reasons, that they want local hardware. The VMs themselves as well as Hyper-V in my test scenario are all running Windows Server 2012 R2.
But what if that hardware breaks? What if there’s a disaster of some sort that makes their server inoperable and their offices uninhabitable? No fun, we have to wait for the hardware to be repaired/replaced, and then do restores. What if we could instead do the restores to a virtual machine in our environment? After all, we do use Veeam Backup and Replication (which is a nice product by the way), and we might already let them store backup copies in our datacenter.
Well, no such luck, because Veeam can’t restore a Hyper-V VM onto vSphere, nor vice versa. Which kind of makes sense, V2V is a hard problem that Veeam doesn’t seem to want to tackle.
We might consider setting up a Windows Server with Hyper-V or two in our datacenter to deal with these potential use cases, but that’s hardware that would just sit unused most of the time. So, that probably won’t happen here.
But all isn’t neccessarilly lost! What if you could use Veeam to Restore VM Files rather than restore the VM itself? That would basically tell Veeam to dump out the constituent files making up a Hyper-V VM, in this case the configuration file (XML) and the VHDX files, and then use some kind of tool to import the Hyper-V VM into VMware from that intermediate restored format? Let’s see.
The first that comes to mind is VMware vCenter Converter. I have that installed on my vCenter box already for other reasons. It looked quite promising, after all, it advertises itself as being able to convert Hyper-V VM:s, but if you look at it a little closer, it seems like it can only do conversions from actual Hyper-V boxes, not from a set of VM files.
This discouraged me a little, and I looked around for other tools. I found (but didn’t try) Starwind V2V Converter, which I probably would have ended up going with, if it weren’t for the fact that it’s a hassle to basically create a new VM and then attach disks to it. There had to be a better way.
That’s when I thought about VMware vCenter Converter again. It has to talk to a Hyper-V server. It has to be registered there, but it also has to be powered off! Aha!
So… what if I would in fact set up a Windows server with the Hyper-V role? I would not have any intention of actually running any VMs on it, just use it to register VMs being restored off Veeam Backup and Replication, for the purposes of VMWare vCenter Converter then doing a V2V of it onto our real production cluster?
I set out to try it. I decided to use my Veeam Backup server (running Windows Server 2012 R2) as the Hyper-V box. It’s already a physical disk with a ton of Storage Spaces JBOD, and I wouldn’t have to send the data over the network for the initial staging of the VM.
So, I decided to start out by simply adding the Hyper-V role to the box.
It asked me about virtual switches, I just skipped that, since I don’t intend to power on any VMs on the box.
It asked me about setting up Live Migration, I skipped that as well, since we’re not doing that.
It asked me about default storage locations. I decided to leave it at the default for now, and see about adding some scratch storage for receiving the restore. For a simple test just doing it on the C-drive where there was ample space was good enough.
After rebooting, I went back into the Veeam Backup & Replication console, and went to Backup Infrastructure and added a new Microsoft Hyper-V Server – the backup server itself. It had to install a few components and took a few minutes.
After that, I made Veeam Backup & Replication restore the entire VM to the backup server where I just installed the Hyper-V role. After the restore was complete, I looked over at the Hyper-V manager on the backup server. Indeed, the VM was there but powered off. I did not attempt to power it on.
Instead, I went over to my vCenter server, where I have vCenter Converter installed. I might just as well have had it on the backup box, but this is where I already had the software. I pressed “Convert Machine”, selected “Hyper-V Server” as a source type, and pointed the software at my backup server.
Here’s where I ran into a hiccup related to Windows Firewall. The diagnostic logs (within two layers of ZIP files) of vCenter Converter showed that it was attempting to connect on port 9089, like so:
2016-02-23T14:09:29.947+01:00 warning vmware-converter-worker[03124] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x0169b440, h:1700, <TCP ‘0.0.0.0:0’>, <TCP ‘192.0.2.123:9089’>>, e: system:10060(A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond)
As far as I can tell this is what happens: If that port shows “Connection refused” then it would offer to install the proper agent on the remote system. If it just timed out (which is what Windows Firewall will do when somebody connects to a port that doesn’t respond, due to Stealth Mode) then it assumes there are network issues and does not offer to deploy the agent.
Advice I found online said to disable the Windows Firewall, but that’s not how I roll. Instead, I created a firewall exception for TCP port 9089 from the vCenter server to my backup server, and then disabled stealth mode in Windows Firewall. There’s no supported or GUI way to do this, you’ll have to dive into the registry.
Following the instructions from this Server Fault answer, I created the registry key HKLM\Software\Policies\Microsoft\WindowsFirewall\DomainProfile, and a DWORD value under it named DisableStealthMode set to “1”. After that, I restarted the Windows Firewall service. (That, or just save yourself the hassle and install the converter on your Hyper-V host.)
After that I was able to proceed to actually deploy the agent in VMware vCenter Converter. Yet another pain in the net…
I chose to manually uninstall the files later, as I anticipate to perform more conversions in the future, but then:
The version of the selected Microsoft Hyper-V Server is not supported. Select an earlier version of Microsoft Hyper-V Server or another supported source type.
More pain. Looking at the release notes, it turned out that I have an older version of VMware vCenter Converter installed – version 6.0 doesn’t seem to support Windows Server 2012 R2 according to the release notes. Fortunately, version 6.1 does seem to support it. So, I proceeded to upgrade it. I also tested re-enabling Stealth mode on the backup server’s Windows firewall to see if the same bug was still present with a recent version. Looks like this bug is still present as of version 6.1.1 build 3533064, which is the most recent as of posting. Oh, VMware, why don’t you test your crap with Windows Firewall enabled? Disabling Windows Firewall may be a common practice, but it’s hardly best practice.
Anyway, now I repeated my steps, and again, I entered the hostname and the credentials, and entered my credentials. And YES! I see my powered off VM I want to get into vSphere.
So, I choose it, and press next, and here comes more pain.
Permission to perform this operation was denied.
Back into the diagnostic logs. This time, I find the appropriate log within three layers of nested ZIP files. Somebody at VMware must have a real fetish for Russian matryoshka dolls, all that nested unzipping… Fortunately, the cause seemed quite simple this time:
2016-02-23T15:08:20.277+01:00 warning vmware-converter-agent[06692] [Originator@6876 sub=Default] UAC is enabled! using non elevated administrator ‘contoso\pv2b-admin’
So, I restart VMware vCenter Converter Standalone again, and this time, elevated as an administrators. Convert machine, powered off, servername, user name password, deploy agent, select VM, next, and again. Permission denied.
This time, I guess the problem is probably Remote UAC being enabled, since it’s trying to get admin over the network. Yet more registry hacks chipping away at security… incidentally Veeam needs the same to work, so I figure I’d try that, according to the solution I found over at Xyfon solutions.
Onto the backup Server, inside the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System, I create a DWORD named LocalAccountTokenFilterPolicy with a Value of 1.
Then I try the same steps again. Nope. Permission still denied. At this point, I get suitably annoyed at VMware’s barely functional client-server software as well as the vagaries of UAC, and decide to just install vCenter Converter Standalone over at the Hyper-V box instead, just to make the netpain stop. If that works, I decide I can just roll back the firewall rule and disabling stealth mode.
Copy installer, paste installer. Wait. Double-click installer. Yes to UAC. Wait while mesmerized by 1990s style installer splash screen. Next, next, agree, next, next, next, next, install. Run standalone client now? Yes. Finish.
Then, I try it again. Convert machine, powered off, Hyper-V server, localhost, username, password, next. Permission denied. Argh. Oh, wait, the installer didn’t launch the converter elevated.
Elevate, repeat steps, still broken. Argh. Curse at the screen for a while, and then realise that I’m stupid. It’s not going to matter if I run the converter itself elevated, because it doesn’t try to use my network creds in a pass-through mode. So, as another stupid simple way, I try using the built-in Administrator account, which because of compatibility reasons (such as stupid applications like this one) does not have UAC enabled.
After that, I’m able to proceed. Seriously, am I really supposed to be crippling the entire security model of Windows just to convert a VM? Anyway, I roll back my registry hacks and firewall rules from above and retry, making sure they’re not neccessary as long as using the built-in Administrator.
After that, I just breeze through the wizard. I make sure the network card is connected to an isolated network, choose some appropriate storage in vSphere, and submit the job, and it seems to be running. Success seems within grasp! Estimated time remaining: 1 hours and 3 minutes. I decide to do some actual work, such as preparing for an after-hours P2V of a dying Windows Server 2003 SBS. (The irony is, we’re already in the process of moving him to a Windows Server 2012 R2 environment, but the server’s so flaky we can’t even get the file share data out of there reliably without his disks timing out… but that’s another article.)
34 minutes later, the VM conversion has finished. I look over in the vSphere Web Client and confirm that the VM is actually on an isolated network. I power on the VM, open the console, and grab figurative popcorn. “Getting devices ready”… hmm… seems there’s some kind of driver injection going on… and then, with no further ado, I get to the login screen. I check the vSphere Client to see if VMware tools is installed – nope. Not surprised not really disappointed, it would have been really cool if vCenter Converter brought them in, but that would just be asking too much.
I log onto the machine with the local admin account. First thing, I look at the network card. It’s there, but set to DHCP. Static IP settings are missing. Not surprising, since it’s a new network card. That’s all right, I just type in some new settings. I type in the same settings as the old VM had, expecting problems with hidden devices. And yes – that’s the case. “Already assigned to another adapter”. I click past it and proceed.
At this point, I’m happy enough to call the restore a success. It boots, and everything important (disks and network) seems to work, as well as the size of the VM (cores, RAM, disk layout etc) being appropriately set as by magic. Very convenient considering. I just kick off an install of VMware tools, making sure that wouldn’t cause any problems, and that’s when I reach some minor netpain:
Call “VirtualMachine.MountToolsInstaller” for object “server01.contoso.com” on vCenter Server “VCENTER” failed.
vix error code = 21002
This virtual machine does not have a CD-ROM drive configured. A virtual machine must have a CD-ROM drive configured for this option to work.
I just hot-add an optical drive into the VM, without rebooting. (Sometimes I love vSphere), and try again. VMware Tools installs like a champ, and I get to reboot to finalize the install. Nice!
Finally, I get to log in to the backup server, open up the Hyper-V manager, and delete the intermediate VM that was registered in Hyper-V. It has served its purpose. Also, can’t remember deleting the VHDX files, which keep the actual data, because deleting the VM does not also delete the virtual hard disks.
So… once this is all set up, this is my workflow for spinning up a Hyper-V VM backed up by Veeam on our vSphere setup:
- Perform a Veeam restore of the Hyper-V VM onto the Hyper-V role on the backup server itself. Use suitable scratch space on the local server.
- Run VMware vCenter Converter, to convert an offline VM from Hyper-V server. Source server localhost, make sure to use the built-in local administrator account (not any old admin account) due to UAC shenanigans.
- Target your vCenter server, and restore onto suitable storage in the cluster.
- Boot up the VM, add a CD-ROM drive, install VMware tools, reboot.
- Set up proper IP addressing, reboot again.
- Test, and bring into production.
- Nuke the intermediate Hyper-V VM on the backup server. Make sure to also delete the associated VHDX files.
Entirely doable, although it involves two copies of the entire VM data, it’s not exactly efficient. But it’s good enough for now. 🙂
One thought on “Cross-hypervisor restores with Veeam and vCenter Converter”