Month: February 2016

Backup is a royal pain in the net. There’s a lot of backup software out there. Some good, some not so good.

Windows Server Backup (or ntbackup) is marginally useful and dangerous at worst. I’ve seen it set up at some small shops, mostly because it’s free, but I have two main pain points with it.

First of all, Windows Server Backup will not back up to drives with 4 KiB sectors. We had a customer who was backing up to a USB drive that failed. We brought in a new USB drive, that had 4 KiB sectors, and it would not back up. Googling found lots of angry people but no solutions. That, and the fact that it’s nigh on impossible to find out if a drive has 4 KiB or 512 B sectors before buying, makes this an ongoing problem.

Let me repeat that, in case that happened to glide past you. The backup destination drive, i.e. where the backup images themselves are stored, i.e. not the drive being backed up, has to have 512 B sectors. This is insane. Why would software ever need to care so deeply about where the files are being stored? There’s probably a limitation saying that the drives to back up also have to be 512 B, but don’t quote me on that. I haven’t tried.

Second, Windows Server Backup has no sensible way of telling you when backup fails. This is insane. You setup a backup solution, it seems to work, then something or other breaks, and it fails, writing an event to an event log that nobody reads often enough to notice. This makes it dangerous, unless you have as a procedure to read your event logs daily at best, or at the very least on a regular basis (not just reading them in the case of a problem). At worst, the problem can persist for months or even years until your main drives actually fail, and you’re caught with your pants down with no recent backup.

Microsoft’s strategy to deal with this piece of suck is to put it out to pasture, pretend it doesn’t exist and not fixing any problems.

Now, I’m told that you can set up alerting on errors with Windows Server Backup in different ways – one way I’ve heard is using Task Scheduler to trigger sending an e-mail on different Event IDs using a script.

The problem is, I don’t trust myself not to screw this up. You basically have to know in advance what log events to be looking for, i.e. enumerate all possible bad conditions, before sending out logs. This is not something individual sysadmins should need to do, that’s the job of whoever makes the software. Besides, I’m not likely to do that kind of work for software that won’t even back up to (i.e. store backup files on!) 4 KiB drives, and that Microsoft considers dead.

So… whenever I see it, I try to get it ripped out and replaced. With what? Well… that’s a good question…

So, yesterday, I had the requirement to shrink a partition. The partition was spanning four dynamic disks, with a total size of 7.5 TiB.

So, using Disk Management, I just go into the GUI, right-click this monster of a partition, click shrink, wait a couple of minutes for it to analyze the free space, and then to tell it to shave of a little north of 4 TB. The goal was to get rid of the two last dynamic disks to reclaim disk space. I fully expected this to take its sweet time. After all, moving data takes time, especially up to 4 TB of it. For the shrink to work, it basically has to pick up all the data past the end of the new partition, and jam it in at the start of the drive.

I was expecting like… a progress indicator which you might expect on any long-running operation? Nope. All I got was a Windows 3.11-style hourglass mouse cursor whenever my mouse is inside Disk Management. Totally useful.

So I leave it overnight, and it’s still running. Curious to see what the hell’s going on, I look at the Resource Monitor. Under Disk, I see that svchost.exe (defragsvc) is reading and writing to my disk at 20 MB/s. Not exactly high speed, but meh. Back-of-the-envelope calculation tells me that 4 TB at 20 MB/s would take 60 hours. That’s fine, it can run over the weekend (I’m not too fussy about how long this takes even though it’s dog slow), but I still wanted to find out its progress. Because I like micromanaging my computers. Like a project manager, I demand percentage-based progress reports!

The fact that it’s defragsvc doing the work is an important clue (and it makes sense too, after all, it is a kind of a full-disk defrag that has to happen). So, I go on the Googles (defragsvc progress), and I click on a random vaguely promising link and end up at the Windows Seven Forums, on a thread labeled “defrag display”. I scroll past most of the stuff that’s not relevant to my case ut then I found this one post listing a few interesting switches to defrag.exe – the command-line interface to defrag. The one that caught my eye was not the one in bold, but rather these two:

/T – Track an operation already in progress on the specified volume.
/U – Print the progress of the operation on the screen.

So… on a whim, I open up a shell and type in “defrag d: /t”. Of course, whoever installed the server made it in Swedish. I have nothing against localized operating systems, in fact I make it a point to make my own PC run in Swedish, but servers need to be installed in English, but I digress. (I’ll probably write something on that.) Anyway, this is the output I got:

PS C:\Users\Administratör> defrag d: /t
Microsoft Diskdefragmenteraren
Copyright (c) 2007 Microsoft Corp.

Spårar åtgärd på Data (D:)...

That says “Tracking operation on Data (D:)”. But there’s no actual other information. I wonder about “what’s the point of tracking an operation if you don’t see any progress?” so I Ctrl+C out of that and instead try:

PS C:\Users\Administratör> defrag d: /u
Microsoft Diskdefragmenteraren
Copyright (c) 2007 Microsoft Corp.

Anropar defragmentering på Data (D:)...

 Analys: 100 % färdigt.

Rapport före defragmentering:

 Volyminformation:
 Volymens storlek = 7,49 TB
 Ledigt utrymme = 4,94 TB
 Totalt fragmenterat utrymme = 5 %
 Storlek på största lediga utrymmet = 599,77 GB

 Obs! Filfragment som är större än 64 MB inkluderas inte i fragmenteringsstatistiken.

Utför pass 1:
 Defragmentering: 0 % färdigt...

So… it actually started a new defragment. I also got a pop-up message from Windows (probably from Disk Management) saying my shrink operation was aborted. Balls. It seems that I told it “defrag drive D, and show me the progress as you go”. Not what I wanted. I Ctrl+C’ed out of that as well. I would have been more annoyed at having the defrag aborted half-way, but the fact is that the subsequent defrag run will just have less to do anyway, and I’m not really in a rush.

So, then I thought, maybe you need both switches? Would make sense, in a perverse Microsoft way. Track the operation and show me the progress.

So, I click through starting the shrink process again, and then I try this:

PS C:\Users\Administratör> defrag d: /t /u
Microsoft Diskdefragmenteraren
Copyright (c) 2007 Microsoft Corp.

Spårar åtgärd på Data (D:)...

 Analys: 100 % färdigt.

Rapport före defragmentering:

 Volyminformation:
 Volymens storlek = 7,49 TB
 Ledigt utrymme = 4,94 TB
 Totalt fragmenterat utrymme = 5 %
 Storlek på största lediga utrymmet = 599,77 GB

 Obs! Filfragment som är större än 64 MB inkluderas inte i fragmenteringsstatistiken.
 Krymp: 53 % färdigt...

That looks promising. “Krymp” is means “shrink” in Swedish. So it’s actually showing me progress of the exising operation. Yay!

I found it interesting that it started out at 53% and not at 0% – maybe it recognised it had been aborted? Or maybe it calculates the progress from the “worst case possible” shrink operation…?

Would be nice with a decimal point or two more, so you can actually stare at it and see it moving. Or maybe a bogus ETA? Oh well, it’s a lot better better than just an hourglass. In the end, the best way for me to actually figure out how long this is going to take, is to wait a few hours, see how many percentage points it’s done in that time, and extrapolate.

So, to recap, if you have a current long-running shrink operation happening that you started from Disk Management, if you want to know what’s going on, pull up a shell, and type in:

defrag X: /t /u

Substituting the drive letter you’re shrinking for X, of course.

As a final word of warning: Do not Ctrl+C out of the progress display! If you do, it’ll actually abort the defrag operation! Which actually makes no sense, since you’re just tracking an ongoing operation… but this is what I’ve observed.

Update: A few hours after originally posting this, I decided to check on it, and see how it was doing. It was still stuck at 53% complete. I killed it and restarted it, and it came to 53% done after a minute or so, jumping there directly from 0%. It may well be that the progress indicator simply does not work for whatever reason, and that this entire post is useless.

Update 2: I discovered the same progress information can be found by opening the Disk Defragmenter UI.

Update 3: One day later, we’re at 54%. I think however that this is due to a very large inbound file transfer (on the order of 1 TB) that’s been running for over a day now to this server, at the same time as trying to shrink. So the progress is moving after all. I expect it to start moving a lot faster as soon as that large file transfer is finished.

All the hallmarks of net pain are right there in the title.

• P2V.
• Microsoft Windows 2003 Small Business Server
• Using CloneZilla.
• To Hyper-V.

First of all, Windows Server 2003 Small Business Server is awful. It’s old, it’s unsupported, and the friendly features work against you more than they work with you. Not to mention that as a final fuck you, it decided that it was not permitted license-wise to do a domain trust to a new forest, further complicating our migration. But that’s another blog post whatsoever.

Second, why am I using CloneZilla instead of some more supported tool? Well… there doesn’t actually seem to be any good P2V solutions to Hyper-V. Microsoft has a P2V tool that’s part of System Center Virtual Machine Manager, which we’re not going to install for a single-hypervisor node. So, that means we’re on our own from Microsoft. I don’t even think their official P2V even supports Windows Server 2003 anyway, but I digress.

The other go-to method for a simple P2V into Hyper-V is using the awesome tool called Disk2vhd. I’ve used this in the past to great success, but I decided not to use this method this time, because the machine I’m cloning from is a domain controller and a mail server. It’s not a good idea to clone these online, because the clone you get will always be stale, and you get “time warp” effects including lost e-mail that might have been delievered while the clone was in progress.

So, I elect to use CloneZilla. I plan on using it in disk_to_remote_disk mode, as documented in this Wiki artickle from Ogala Lakota College. Last edited in December 2009. So the article is over six years out of date already. I feel the pain already…

My plan is to run the CloneZilla live CD on the Hyper-V box and use that to receive the clone, after booting the physical box with a burned CD of the ISO file of CloneZilla. No sense risking flaky USB booting on a machine of that vintage. My fall-back plan is to isolate the server from the network, and use Disk2vhd after all, which should in theory be fine as long as the server doesn’t talk to the network.

I download the ISO and for a moment I have to double check that the old creaky hardware actually supports 64-bit OS’es. Turns out it does. Armed with Plan A and half of a Plan B, I pack my backpack for the evening.

---

Arrived at the customer site, armed with the CD. I boot the machine with it. Notice the RAID array at the customer is degraded with both disks healthy. Maybe that had been causing some issues? Never mind, that’s exactly why I wanted to P2V.

After some ominous read errors on the CD drive, CloneZilla live booted without a hitch. Actually, the instructions seemed to be entirely up to date despite being written in 2009. Yay for stable software.

At some point, I’m presented with a choice between two hard drives. Hmm. Seems CloneZilla isn’t fooled by the server’s FakeRAID and presents the two drives seperately. Good thing it’s a RAID-1. Also, the array was degraded anyway, so I knew one of the two drives had to be stale.

I pressed Alt+F2, entering into the command shell, become root, and examine the situation using fdisk. Both drives have two NTFS partitions. (fdisk /dev/sda , then p to print the partition table, then q to exit without making changes)

I decide to look a little deeper to see which one’s recent and which one is stale. I mount the sda1 and sdb1 partitions read-only using these commands:

cd /mnt
mkdir a b
mount -t ntfs -o ro /dev/sda1 /mnt/a
mount -t ntfs -o ro /dev/sdb1 /mnt/b

After a bit of poking around and looking at file dates, it’s apparent that the copy on /dev/sdb is up-to-date. The other one is quite stale.

At that point I unmount the two partitions:

umount a
umount b

And return back to the menu’s using Alt+F1. Certain of my choice, I choose to clone from /dev/sdb

After following through, it gives me a few commands I need to type into the VM where I had booted CloneZilla before. The source machine is saying “Waiting for the target machine to connect…”

I enter the commands on the target machine, and a progress window shows up on the target machine! Success! Although there is no indication that there’s any file copy happening on the source machine… oh well. It’s definitely copying.

It asked a few questions regarding whether to copy over the boot loader etc, I just said yes.

I leave and come back to a progress bar at 90%. I watch it go to 100%, and then I realised I’d been trolled. The progress bar is volume by volume. Now, I still have to wait for the second volume to finish copying… all the while standing in the hot and untidy server closet at the customer, with nowhere to sit. I text my wife I expect to leave in about an hour 45 minutes – around 20.00 – after all, there’s another 22 minutes left on the file copy, and I’ll probably need just north of an hour to actually get the server to work right. Faith is high.

At this point, I’m waiting patiently, expecting the VM to come back up after reboot, probably screaming bloody murder about drivers, but in general being in a fixable state. I hope. 5 more minutes. I wait.

I write a hand-written note. “This machine – SRV03 – HP Proliant ML110 – must NOT BE STARTED”. Best to cover all the bases since the old server is not labeled with a hostname. I plan on unplugging the power from the server as soon as the copy has finished. Better safe than sorry with two almost-same but different SBS servers duking it out on the network in a Small Business Deathmatch. I duct tape the label over the power button. Oooh, copy complete. I restart the virtual machine after disconnecting the virtual CD-ROM, and I’m met by – nothing. A blinking cursor on a black screen.

At this point, I decide to spend a few minutes to find a way to sit down, hook up my laptop and get comfortable. This was going to take some effort.

After a few seconds of Googling (windows 2003 sbs hyper-v black screen cursor), I find a Technet blog article titled VM does not Boot Following P2V or Disk2VHD

Looks like I’m going to need to get Windows Server 2003 boot media. No problem. I grab an ISO off that off the Volume Licensing Service Center. Or so I thought. Microsoft no longer has ISO’s of that on VLSC. Meh. Figures though. I should have been better prepared and got an ISO of that onto the virtualization host just in case. I’m not surprised Microsoft has pulled the ISO’s, the product is no longer under support after all.

I VPN into our datacenter on a quest to locate a Windows Server 2003 ISO. We’re bound to have one laying around. For the first time ever I find the web-based datastore browser to actually be useful. Rooting around in a musty old folder named “From old datastore” I find a copy of Windows Server 2003 R2 32-bit. I’m pretty sure the server was running 32-bit to start, since I saw that it had PAE enabled. It’s not SBS, but for a bootup fix it shouldn’t matter.

I download the files, and then copy them up to the new shiny Hyper-V box. I spend a few minutes admiring progress bars, and mentally berating myself for not thinking of this ahead of time. Mental note: Always make sure there’s a copy of the install ISO for the P2V:ed or V2V:ed operating system ahead of the actual V2V. Better have it copied ahead of time instead of doing it over a VPN in an uncomfortably hot server closet.

I insert the virtual disc into the virtual DVD drive and reboot, and I see the windows installer. I get nostalgic for a moment as the old text-based pre-installer shows up.

I press R for repair, select my keyboard, select the windows install, type in the admin password, and then enter the magic incantations, and confirming along the way:

fixmbr
fixboot
bootcfg /rebuild
exit

I get up two possible boot options after rebooting, Windows Server 2003 for Small Business Server, and my new option I labeled “P2V”. I decide to choose the old option to see if it works. It’s not clean to have two options, so I plan on manually editing C:\boot.ini to clean it up, once I’ve established which one works. Then, I see the old familiar Windows Server 2003 boot screen. I wait.

The GUI loads, preparing network connections. I realise I’ve forgotten to note down the old IP address. No matter, I log on to their old firewall to look at the DHCP scope to find the DNS server, that’ll be it. Stupid. Should have known that setting would get reset, I just did another V2V a few hours ago.

It reaches the login screen, running at 640×480 resolution with no mouse support. Yay. Fortunately, I’m pretty good with keyboard navigation in windows. I insert the Hyper-V integration services disc, log in and make it the first order of business to install the integration services.

Windows wants to activate because of major hardware changes. I tell it to do it later, since it’s not likely to have working networking. At this point I’m ready to declare that I will be successful, so I shut off the source machine (which was still running CloneZilla), and unplug it from the power. I decide to also unplug it from the network. You can never be too safe.

The Hyper-V integration services finish installing, and it asks me to reboot. And the mouse actually works \o/ Faith is quite high. I reboot the server again.

The old crusty SBS takes quite some time to reboot, but it gets there in the end. That’s all the matters. I log in, and make sure to set the correct static IP. Suddenly, Microsoft gets all agressive on me and wants to activate NOW NOW NOW or it’ll shut down. I decide to let it, it should be able to get out to the Internet through a DHCP address. Activation passes without a hitch. Good, last thing I wanted was activation issues.

I investigate the VM config. Really? Only one CPU core? The old physical server had a dual-core Xeon (oooh!), at least it should have two cores… I plan on shutting down and adding another processor core after changing the IP address.

Wait a moment, where’s my explorer? After activating, all I got was a blue desktop background screen (not to be confused with a BSOD). I wait for a few minutes, still nothing. I hit Ctrl+Alt+Del, bring up a task manager, nothing’s taking the CPU. I decide to try to kill and re-launch the explorer manually. Nothing. Hmm. I launch cmd. That works. I decide to log out and then log back in. Still nothing. I decide to try a shut down. Might as well add that second CPU core now. Remembering from the back of my head that going between a single-processor and a multi-processor OS install can screw with things. Not sure if that’s applicable here.

I look at my watch. Half an hour until I estimated I’d get out of here. Looking a little unlikely but not impossible.

Realize after rebooting that the new boot config was the default when booting. Hmm. Maybe that’s a relevant difference? Remind myself again to tweak boot.ini once I get the IP settings right.

Phew. After a reboot, explorer pops up just like it should after reboot. Might have been either a bug with the activation, or with the wrong number of CPU’s, or the boot.ini. Don’t know, don’t care right now. 🙂

When changing the IP, I get warned about the same IP being present on another network adapter. I ignore that for now, making a note to remove the hidden network adapters.

I log on to the server through RDP rather than through the console. Not being stuck at 640×480 is a liberating experience.

I adjust the boot.ini, removing the redundant section from the boot configuration. I find the following in the config:

[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="P2V" 
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003 for Small Business Server" /noexecute=optout /fastdetect

This tells me it couldn’t possibly have been the two different boot options, since they both just do the same thing. I remove the line with P2V in it anyway. Boot.ini wouldn’t save in notepad, until I did this in CMD:

attrib -s -h -r c:\boot.ini

Then I saved, and I did:

attrib +s +h +r c:\boot.ini

Another reboot, to make sure everything seems to stick. I decide to shut it down using the integrations services to make sure that works right. It does.

I check my watch. 19.45. Looks like I’ll be able to leave when I expected after all! Or at least not too far off. I start packing my stuff, and type in the details in my phone to order an Uber home.

The server reboots, is RDP:able, I check the event log, nothing unexpected, looks good. Server sure is snappier than on the old hardware. Customer’s probably gonna like that. I shut down my computer and head home.

On the way home, I text the customer to call me in the morning, if anything’s broken. I plan on sleeping unmolested, though.

On-site off-hours work is just a pain in the net. But it could have been worse.

 

Let’s say I have my own hosting platform based on vSphere, where I run most of my customers, as a service provider, on a decent cluster with multiple nodes and shared storage and all that jazz.

Let’s further say that I have customers that have local Hyper-V environments running on single boxes here and there, because those customers have chosen, because Reasons, that they want local hardware. The VMs themselves as well as Hyper-V in my test scenario are all running Windows Server 2012 R2.

But what if that hardware breaks? What if there’s a disaster of some sort that makes their server inoperable and their offices uninhabitable? No fun, we have to wait for the hardware to be repaired/replaced, and then do restores. What if we could instead do the restores to a virtual machine in our environment? After all, we do use Veeam Backup and Replication (which is a nice product by the way), and we might already let them store backup copies in our datacenter.

Well, no such luck, because Veeam can’t restore a Hyper-V VM onto vSphere, nor vice versa. Which kind of makes sense, V2V is a hard problem that Veeam doesn’t seem to want to tackle.

We might consider setting up a Windows Server with Hyper-V or two in our datacenter to deal with these potential use cases, but that’s hardware that would just sit unused most of the time. So, that probably won’t happen here.

But all isn’t neccessarilly lost! What if you could use Veeam to Restore VM Files rather than restore the VM itself? That would basically tell Veeam to dump out the constituent files making up a Hyper-V VM, in this case the configuration file (XML) and the VHDX files, and then use some kind of tool to import the Hyper-V VM into VMware from that intermediate restored format? Let’s see.

The first that comes to mind is VMware vCenter Converter. I have that installed on my vCenter box already for other reasons. It looked quite promising, after all, it advertises itself as being able to convert Hyper-V VM:s, but if you look at it a little closer, it seems like it can only do conversions from actual Hyper-V boxes, not from a set of VM files.

This discouraged me a little, and I looked around for other tools. I found (but didn’t try) Starwind V2V Converter, which I probably would have ended up going with, if it weren’t for the fact that it’s a hassle to basically create a new VM and then attach disks to it. There had to be a better way.

That’s when I thought about VMware vCenter Converter again. It has to talk to a Hyper-V server. It has to be registered there, but it also has to be powered off! Aha!

So… what if I would in fact set up a Windows server with the Hyper-V role? I would not have any intention of actually running any VMs on it, just use it to register VMs being restored off Veeam Backup and Replication, for the purposes of VMWare vCenter Converter then doing a V2V of it onto our real production cluster?

I set out to try it. I decided to use my Veeam Backup server (running Windows Server 2012 R2) as the Hyper-V box. It’s already a physical disk with a ton of Storage Spaces JBOD, and I wouldn’t have to send the data over the network for the initial staging of the VM.

So, I decided to start out by simply adding the Hyper-V role to the box.

It asked me about virtual switches, I just skipped that, since I don’t intend to power on any VMs on the box.

It asked me about setting up Live Migration, I skipped that as well, since we’re not doing that.

It asked me about default storage locations. I decided to leave it at the default for now, and see about adding some scratch storage for receiving the restore. For a simple test just doing it on the C-drive where there was ample space was good enough.

After rebooting, I went back into the Veeam Backup & Replication console, and went to Backup Infrastructure and added a new Microsoft Hyper-V Server – the backup server itself. It had to install a few components and took a few minutes.

After that, I made Veeam Backup & Replication restore the entire VM to the backup server where I just installed the Hyper-V role. After the restore was complete, I looked over at the Hyper-V manager on the backup server. Indeed, the VM was there but powered off. I did not attempt to power it on.

Instead, I went over to my vCenter server, where I have vCenter Converter installed. I might just as well have had it on the backup box, but this is where I already had the software. I pressed “Convert Machine”, selected “Hyper-V Server” as a source type, and pointed the software at my backup server.

Here’s where I ran into a hiccup related to Windows Firewall. The diagnostic logs (within two layers of ZIP files) of vCenter Converter showed that it was attempting to connect on port 9089, like so:

2016-02-23T14:09:29.947+01:00 warning vmware-converter-worker[03124] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x0169b440, h:1700, <TCP ‘0.0.0.0:0’>, <TCP ‘192.0.2.123:9089’>>, e: system:10060(A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond)

As far as I can tell this is what happens: If that port shows “Connection refused” then it would offer to install the proper agent on the remote system. If it just timed out (which is what Windows Firewall will do when somebody connects to a port that doesn’t respond, due to Stealth Mode) then it assumes there are network issues and does not offer to deploy the agent.

Advice I found online said to disable the Windows Firewall, but that’s not how I roll. Instead, I created a firewall exception for TCP port 9089 from the vCenter server to my backup server, and then disabled stealth mode in Windows Firewall. There’s no supported or GUI way to do this, you’ll have to dive into the registry.

Following the instructions from this Server Fault answer, I created the registry key HKLM\Software\Policies\Microsoft\WindowsFirewall\DomainProfile, and a DWORD value under it named DisableStealthMode set to “1”. After that, I restarted the Windows Firewall service. (That, or just save yourself the hassle and install the converter on your Hyper-V host.)

After that I was able to proceed to actually deploy the agent in VMware vCenter Converter. Yet another pain in the net…

I chose to manually uninstall the files later, as I anticipate to perform more conversions in the future, but then:

The version of the selected Microsoft Hyper-V Server is not supported. Select an earlier version of Microsoft Hyper-V Server or another supported source type.

More pain. Looking at the release notes, it turned out that I have an older version of VMware vCenter Converter installed – version 6.0 doesn’t seem to support Windows Server 2012 R2 according to the release notes. Fortunately, version 6.1 does seem to support it. So, I proceeded to upgrade it. I also tested re-enabling Stealth mode on the backup server’s Windows firewall to see if the same bug was still present with a recent version. Looks like this bug is still present as of version 6.1.1 build 3533064, which is the most recent as of posting. Oh, VMware, why don’t you test your crap with Windows Firewall enabled? Disabling Windows Firewall may be a common practice, but it’s hardly best practice.

Anyway, now I repeated my steps, and again, I entered the hostname and the credentials, and entered my credentials. And YES! I see my powered off VM I want to get into vSphere.

So, I choose it, and press next, and here comes more pain.

Permission to perform this operation was denied.

Back into the diagnostic logs. This time, I find the appropriate log within three layers of nested ZIP files. Somebody at VMware must have a real fetish for Russian matryoshka dolls, all that nested unzipping… Fortunately, the cause seemed quite simple this time:

2016-02-23T15:08:20.277+01:00 warning vmware-converter-agent[06692] [Originator@6876 sub=Default] UAC is enabled! using non elevated administrator ‘contoso\pv2b-admin’

So, I restart VMware vCenter Converter Standalone again, and this time, elevated as an administrators. Convert machine, powered off, servername, user name password, deploy agent, select VM, next, and again. Permission denied.

This time, I guess the problem is probably Remote UAC being enabled, since it’s trying to get admin over the network. Yet more registry hacks chipping away at security… incidentally Veeam needs the same to work, so I figure I’d try that, according to the solution I found over at Xyfon solutions.

Onto the backup Server, inside the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System, I create a DWORD named LocalAccountTokenFilterPolicy with a Value of 1.

Then I try the same steps again. Nope. Permission still denied. At this point, I get suitably annoyed at VMware’s barely functional client-server software as well as the vagaries of UAC, and decide to just install vCenter Converter Standalone over at the Hyper-V box instead, just to make the netpain stop. If that works, I decide I can just roll back the firewall rule and disabling stealth mode.

Copy installer, paste installer. Wait. Double-click installer. Yes to UAC. Wait while mesmerized by 1990s style installer splash screen. Next, next, agree, next, next, next, next, install. Run standalone client now? Yes. Finish.

Then, I try it again. Convert machine, powered off, Hyper-V server, localhost, username, password, next. Permission denied. Argh. Oh, wait, the installer didn’t launch the converter elevated.

Elevate, repeat steps, still broken. Argh. Curse at the screen for a while, and then realise that I’m stupid. It’s not going to matter if I run the converter itself elevated, because it doesn’t try to use my network creds in a pass-through mode. So, as another stupid simple way, I try using the built-in Administrator account, which because of compatibility reasons (such as stupid applications like this one) does not have UAC enabled.

After that, I’m able to proceed. Seriously, am I really supposed to be crippling the entire security model of Windows just to convert a VM? Anyway, I roll back my registry hacks and firewall rules from above and retry, making sure they’re not neccessary as long as using the built-in Administrator.

After that, I just breeze through the wizard. I make sure the network card is connected to an isolated network, choose some appropriate storage in vSphere, and submit the job, and it seems to be running. Success seems within grasp! Estimated time remaining: 1 hours and 3 minutes. I decide to do some actual work, such as preparing for an after-hours P2V of a dying Windows Server 2003 SBS. (The irony is, we’re already in the process of moving him to a Windows Server 2012 R2 environment, but the server’s so flaky we can’t even get the file share data out of there reliably without his disks timing out… but that’s another article.)

34 minutes later, the VM conversion has finished. I look over in the vSphere Web Client and confirm that the VM is actually on an isolated network. I power on the VM, open the console, and grab figurative popcorn. “Getting devices ready”… hmm… seems there’s some kind of driver injection going on… and then, with no further ado, I get to the login screen. I check the vSphere Client to see if VMware tools is installed – nope. Not surprised not really disappointed, it would have been really cool if vCenter Converter brought them in, but that would just be asking too much.

I log onto the machine with the local admin account. First thing, I look at the network card. It’s there, but set to DHCP. Static IP settings are missing. Not surprising, since it’s a new network card. That’s all right, I just type in some new settings. I type in the same settings as the old VM had, expecting problems with hidden devices. And yes – that’s the case. “Already assigned to another adapter”. I click past it and proceed.

At this point, I’m happy enough to call the restore a success. It boots, and everything important (disks and network) seems to work, as well as the size of the VM (cores, RAM, disk layout etc) being appropriately set as by magic. Very convenient considering. I just kick off an install of VMware tools, making sure that wouldn’t cause any problems, and that’s when I reach some minor netpain:

Call “VirtualMachine.MountToolsInstaller” for object “server01.contoso.com” on vCenter Server “VCENTER” failed.
vix error code = 21002
This virtual machine does not have a CD-ROM drive configured. A virtual machine must have a CD-ROM drive configured for this option to work.

I just hot-add an optical drive into the VM, without rebooting. (Sometimes I love vSphere), and try again. VMware Tools installs like a champ, and I get to reboot to finalize the install. Nice!

Finally, I get to log in to the backup server, open up the Hyper-V manager, and delete the intermediate VM that was registered in Hyper-V. It has served its purpose. Also, can’t remember deleting the VHDX files, which keep the actual data, because deleting the VM does not also delete the virtual hard disks.

So… once this is all set up, this is my workflow for spinning up a Hyper-V VM backed up by Veeam on our vSphere setup:

1. Perform a Veeam restore of the Hyper-V VM onto the Hyper-V role on the backup server itself. Use suitable scratch space on the local server.
2. Run VMware vCenter Converter, to convert an offline VM from Hyper-V server. Source server localhost, make sure to use the built-in local administrator account (not any old admin account) due to UAC shenanigans.
3. Target your vCenter server, and restore onto suitable storage in the cluster.
4. Boot up the VM, add a CD-ROM drive, install VMware tools, reboot.
5. Set up proper IP addressing, reboot again.
6. Test, and bring into production.
7. Nuke the intermediate Hyper-V VM on the backup server. Make sure to also delete the associated VHDX files.

Entirely doable, although it involves two copies of the entire VM data, it’s not exactly efficient. But it’s good enough for now. 🙂

Licensing is a pain. Not so much for the sake of cost, but more for the sake of never quite knowing that you’re compliant.

I was considering a case today. What if one of my customers wants to grant administrative access to a vendor to one of their servers for the purposes of setting up and administering an on-premise application?

My first instinct was yes – they do. After all, they are a user that needs to log in. But then I thought a bit more about it. CALs are after all assigned to physical persons or devices, I can’t just go say “I hereby assign this CAL to Vendor X’s support team”, because that’s not a physical person. Would I have to ask for specific names of the vendor’s technicians for the purposes of license assignment? And how would this be enforced? Maybe just creating a few named users and then just disclaiming knowledge of whether they share their personal accounts, making it their problem? No. There had to be some kind of exception.

So I go out on the Internet, and I found this slightly helpful but mostly unhelpful post from the Microsoft Volume Licensing Blog. As witnessed by the comments, the blog entry raises more questions than it answers. It’s not really the fault of the blog, licensing is hard, really for no good reason other than to make work for licensing lawyers.

This is what they write:

7 – Do I need CALs for my administrators?

Server software licensed using CALs permits up to 2 users or devices to access the server software for the purposes of administration without CALs. However, if your administrators also use the software for anything other than administration (for example, they check their email), CALs will be required for them as well.

You could be forgiven to still fret and say “Oh, okay, so if my vendor has 10 employees I only need 8 CALs” and open your pocketbook and still have the administrative nightmare about having to know about who they employ and manage CAL assignment, adding the two “free” CALs to your pool of CALs.

But that’s actually not the case. If you read what Microsoft said, they specifically say “Server software licensed using CALs permits up to 2 users or devices to access the server software for the purposes of administration without CALs.” (emphasis mine).

Why is this important? Well, because the rules regarding CAL assignments do not come into play! The only way I could reasonably interpret this would be “permits up to 2 users or devices at the same time“. After all, there is no permanent assignment of CALs going on.

The relevant language from Microsoft’s Product Terms does not contradict this interpretation:

16. Administrative and Support Rights
Customer may allow access to server software running in any permitted OSE by two users without CALs solely for administrative purposes. Customer may also allow remote access to other Products solely for purposes of providing technical product support to Licensed Users or on Licensed Devices.

So, my conclusion is, I’m okay. We don’t need to assign CALs ever to users who only work as admins on the server, as long as no more than two of them are ever working on the same time on any given server.

Now, I’d like to point out for the record. I’m not a lawyer, I’m not a licensing expert. I don’t work for a licensing company.

This is not licensing advice. This is just my opinion about how I interpret the licensing rules. After all, I’m just a sysadmin annoyed with the various “pain in the nets” and blogging about it.

And I find that this obfuscated interpretation lines up with common sense. So that’s the interpretation I’m going with.