Category: Uncategorized

When you buy an HPE ProLiant server, it comes with HPE iLO (Intelligent Lights-out) by default. iLO is a remote management solution, allowing you to remotely control your server, allowing for many actions, including turning the server on and off, remotely accessing the server’s keyboard, mouse and display (as if you were plugged in directly to the server), or mounting (and booting off) ISO files. In the industry, such functionality is generally also known as IPMI, named after the IPMI standard which provides a vendor-neutral standard for this type of system management.

This is a very convenient feature of course for remote management. But it’s also important that the feature is properly secured. Else, the feature can be abused. For example, an attacker could turn your server off, or even restart it with a malicious ISO file mounted, and install malware on your machine. One example of such attacks were the 2018 attacks by a group called JungleSec, which installed ransomware on poorly secured servers through similar features on competing vendors, some of which allow logins using well-known default credentials.

Fortunately, HPE iLO is a cut above many of the competitors offering IPMI solutions, in that HPE will randomize a unique password for every server for the iLO. So, even if you never touch the iLO funcitionality, you should be reasonably well protected, right?

That’s a reasonable assumption to make, unfortunately, it doesn’t turn out to be true in all cases.

IPMI isn’t just a generic name for iLO-like functionality, it’s also a vendor-neutral standard. Unfortunately, baked right into the standard is a pretty glaring design flaw, which is present in all compliant implementations.

I’m not going to get into the weeds with juicy details. All you need to know for the purpose of this post, is that there’s a flaw in the IPMI 2.0 specification, that allows an attacker to obtain password hashes for any IPMI account, as long as they know the username for it. The vulnerability is baked into the IPMI specification, and cannot be patched.

So, therefore, if you can talk to the IPMI service of an HPE ProLiant server, and you know it’s username, you can dump out a hash of the password for the iLO service.

Unfortunately, the username on all HPE ProLiant servers is the same, and is well known, so getting a copy of the password hash is pretty easy, allowing for easy password cracking.

Even more unfortunately, the password generation algorithm used by HPE to set their iLO passwords, do not produce passwords strong enough to resist cracking. In my own tests experiments, I was able to reliably crack a password in less than 30 minutes using a contemporary gaming desktop computer (Geforce RTX 3070, 1600 MH/s), with no more information than what pattern HPE uses for their passwords. A sophisticated attacker might use a more powerful machine, major cloud vendors provide machines with 10x the cracking horsepower for a very practically affordable $25 per hour, making cracking a single HP iLO password cost less than $1 in computing power.

For your server to be vulnerable to this particular attack, a few things have to line up:

1. iLO must be reachable on your network. On some servers, iLO will piggyback on your main network card, and use DHCP. You might have iLO on your network and not even know it. On other servers, iLO only runs on a seperate dedicated network card.
2. You must be able to reach it on UDP port 623, which is the port used by IPMI. As far as I know, it’s not possible to exploit this if only the web interface is exposed. Your firewall may protect you from attacks from remote networks, but unfortunately, a common practice is to put the server directly on your office LAN, allowing an attacker to pivot from a compromized PC to attack your server, which would bypass your firewall. For protection on the LAN side, you may be able to leverage Acess Control Lists on your switches.
3. The IPMI service must be enabled. This setting (called IPMI/DCMI over LAN) is fortunately disabled by default on modern servers (Gen10 and newer), but is ENABLED on older hardware (Gen9 and older).
4. The username must be guessable. On HPE servers, the default username is common across all servers.
5. The password must be crackable. The default passwords used by HPE are crackable using affordable hardware in a short amount of time.
6. Access to iLO must allow you to compromize the host system in a meaningful way. This is almost always true unless you have a robust security configuration.

In order to mitigate this, you need to do at least ONE of the following things. You don’t need to do everything, of course, and you probably don’t want to do everything on this list. What mitigations you end up putting in place will depend on your specific requirements.

1. If you don’t need iLO, consider completely disabling the functionality. Personally, I wouldn’t do this, because iLO is legitimately far too useful to disable. You should still use firewalls and network segmentation to make sure only specific, trusted computers can reach the iLO interface.
2. If you don’t want to lock down access to the iLO web interface too hard, you can still protect yourself by locking down access to UDP port 623. That will stop this attack in its tracks, even if the web interface is accessible. Remember, your firewall will not protect your ILO card from attackers on the same LAN.
3. If you don’t need access to industry-standard (and unsecure) IPMI, consider disabling IPMI/DCMI over LAN entirely. That will also stop this attack in its tracks, and you can still manage iLO using the web interface and through HPE’s own tools.
4. If you must have a weak password, consider changing the username, and guard the username as a close secret. The username is required to do a dump of the password hashes through IPMI.
5. Use a long, secure password for iLO. I would suggest at least a 15-character password containing random uppercase, lowercase, and digits.
6. If you really must run IPMI, accessable from untrusted devices, and you really must have a weak guessable username, and a crackable password, your last line of defense would be implementing BIOS passwords to prevent an attacker from booting off removable media. You will also need to be aggressive about locking your screen and/or logging off from the console when the server is not in use. Even in these cases, it’ll be possible to remotely shut down your server. It’s not clear how effective this would be in the long run to stop a determined attacker, but it’s worth considering as a last line of defense.

Finally, I’d like to point out that the ability to dump passwords out of IPMI boxes is not news. Dan Farmer wrote about this in 2013. But even though it’s not news, it’s still a current vulnerability, and because the vulnerability is in the standard, we just have to learn to live with it.

What I feel I bring to the table with this article is to raise awareness of this bug, as well as HPE’s default passwords are not strong enough to resist this type of attack, and in fact are crackable in minutes. I’ve always been impressed that HPE goes above their competition and set strong passwords by default, unlike other vendors, but unfortunately, they don’t stand up to this kind of attack. To HPE’s credit, modern servers have IPMI over LAN disabled by default, and modern servers will also strongly encourage you to change default passwords.

Because my mission is to help server administrators defend their systems, not to write a HOWTO for would-be attackers, I will not be provide details as to default usernames used by iLO, or the specific pattern used by HPE to generate passwords. This information is not useful to defenders, and would only serve to help attackers. I will also not name the specific tools used to pull off this attack, but they are publically and readily available with a little bit of research. It’s not difficult to get this information elsewhere, but I will not be the one providing it to you, nor will I approve any comments requesting or providing such information.

Stay safe out there!

References:

A Penetration Tester’s Guide to IPMI and BMCs (HD Moore, Rapid 7, Jul 2 2013)

Cracking IPMI Passwords Remotely (Dan Farmer, fish2.com, date unknown, circa 2013)

HPE Security Bulletin HPSBHF02981 rev.4 – HPE Integrated Lights-Out 2, 3, 4, 5 (iLO 2, iLO 3, iLO 4, and iLO 5) and HPE Superdome Flex RMC – IPMI 2.0 RCMP+ Authentication Remote Password Hash Vulnerability (RAKP) (published May 2018)

JungleSec Ransomware Infects Victims Through IPMI Remote Consoles (Lawrence Abrams, BleepingComputer.com, 26 December 2018)

Ever wanted to reboot a Windows box at a specific time? How hard could it be? Unforunately way too hard.

The Restart-Computer cmdlet in Powershell unfortunately does not seem to have an option to schedule a for a later time or delay a restart. The old school “shutdown” command, does though. Unfortunately:

    /t xxx     Set the time-out period before shutdown to xxx seconds.
               The valid range is 0-315360000 (10 years), with a default of 30.
               If the timeout period is greater than 0, the /f parameter is
               implied.

The functionality is there but it’s a pain to use. I don’t want to have to do math to calculate the number of seconds until shutdown. That’s why we have computers after all!

PowerShell to the rescue after all! Consider the following example:

shutdown /r /t ([math]::ceiling((New-TimeSpan -End "2016-11-15 22:00").TotalSeconds)) /d p:1:1

This example will restart the server at 22:00 on 2016-11-15, with the reason of “Hardware: Maintenance (Planned)”.

Picking apart the command

As before, I like to take apart the command to show what makes it tick.

The command shutdown will call the “old school” shutdown command found at C:\Windows\System32\Shutdown.exe.

The /r parameter specifies that we want to do a reboot.

The /t parameter specifies that the next parameter is the number of seconds until restart.

The next parameter, ([math]::ceiling((New-TimeSpan -End "2016-11-15 22:00").TotalSeconds)) calculates the number of seconds until the desired shutdown time. We’ll take a deeper look at this further down.

The /d p:1:1 parameter specifies the shutdown code. For a full list of valid reason codes, run shutdown /?.

Picking apart that magic parameter

All the magic of calculating the number of seconds happens here:

([math]::ceiling((New-TimeSpan -End "2016-11-15 22:00").TotalSeconds))

In order to better understand this, let’s unpack the full command into a longer script.

# Generate a TimeSpan measuring the amount of time between now
# (the default when the "Start" parameter is omitted) and the
# end time (the desired shutdown time)
$timespan = New-TimeSpan -End "2016-11-15 22:00"

# Get the amount of time as seconds. We use TotalSeconds here
# and not Seconds, because Seconds only contains the seconds
# part of the time span. E.g. if the time span is 10 minutes
# 30 seconds, Seconds would be 30, and TotalSeconds would be
# 630.
$seconds = $timespan.TotalSeconds

# Unfornately, $seconds contains a lot of unneccessary decimals.
# Here, we're rounding it up to the nearest integer.
$seconds = [math]::ceiling($seconds)

# Finally, we're invoking the shutdown command with the desired
# parameters.
shutdown /r /t $seconds /d p:1:1

I hope this is clear. Happy rebooting!

Today, I had an interesting day with PowerShell, debugging a script.

Consider the following attempt at solving the classic Fizz Buzz puzzle. (Yes, it’s written in a slightly unusual style, but not obviously incorrect.) Can you spot the error?

function Get-FizzBuzz($max) {
    1..$max | Foreach-Object {
        if ($_ % 15 -eq 0) {
            "fizzbuzz"
            continue
        }
        if ($_ % 3 -eq 0) {
            "fizz"
            continue
        }
        if ($_ % 5 -eq 0) {
            "buzz"
            continue
        }
        $_
    }
}

Get-FizzBuzz 20
"All done!"

The output of this script is:

1
2
fizz

And this is my reaction:

Not only did the loop end earlier than expected. But I never even got “All done!” out on my screen. Why would the script just stop half-way? There’s no obvious reason.

Except once you focus in on the fact that Foreach-Object is not a language construct, where continue or break makes any sense. It’s a cmdlet, and the stuff between Foreach-Object { … } is a ScriptBlock that is executed for every object.

It turns out that, unlike in every programming language I’ve worked with before, a “continue” and a “break” is legal, even if you’re not actually directly inside a loop. But in PowerShell, you can break and continue in a loop that’s in a function that you’re being called from!

This is really astonishing, because that simply should not be possible in a structured programming language. Not even C allows for a called function to mess with the control flow of a loop (unless the called function does some low-level trickery to the stack).

So, what you’re supposed to do, is that if you use Foreach-Object, instead of “continue” you can use “return”, and instead of “break”… well… you’re SOL.

This is described for example in this slightly mis-titled Stack Overflow question: Why does continue behave like break in a Foreach-Object?.

I did spend some time on the Virtual PowerShell Group discussing the implications of this. @larryweiss linked me a blog post on pluralsight.com by Adam Bertram named PowerShell: Tips for terminating code execution, which advocates for using “break” to quit a script prematurely (emphasis mine):

Break can also be used outside of its typical constructs, and can simply be placed in a script outside of a loop. In this case, break will stop all script execution, even if called within a function in the script. It’s a great way to ensure the entire script halts wherever it may be (excluding in a loop or switch statement).

To me, that sounds like he’s making a pretty poor case for using it, after all, you can never know as the author of a function if you’re being called inside a loop or not.

To further illustrate this language feature, consider this (crazy but functional) implementation of FizzBuzz:

function Get-Fizz($i) {
    if ($i % 3 -eq 0) {
        "fizz"
        continue
    }
}

function Get-Buzz($i) {
    if ($i % 5 -eq 0) {
        "buzz"
        continue
    }
}

function Get-FizzBuzz($i) {
    if ($i % 15 -eq 0) {
        "fizzbuzz"
        continue
    }
}

for ($i = 1; $i -lt 20; $i++) {
    Get-FizzBuzz $i
    Get-Fizz $i
    Get-Buzz $i
    $i
}

Notice how the Get-FizzBuzz, Get-Fizz and Get-Buzz functions actually control the flow of the for loop in the main script!

We scratched our heads a little, trying to come up with a legitimate use case for this language feature. That’s when I came up with this:

function Foreach-ObjectImproved([ScriptBlock]$sb) {
    Begin {
        $did_break = $false
    }
    Process {
        if (-not $did_break) {
            for ($i = 0; $i -lt 1; $i++) {
                $did_breakOrContinue = $true # Being set speculatively...
                Invoke-Command $sb -ArgumentList $_
                # Will not reach here if Invoke-Command did break or continue...
                $did_breakOrContinue = $false
            }
            # If the ScriptBlock called "break" to break out of the the loop above, $i will be at 0, because the loop-increment and loop-test will not have run.
            # If the ScriptBlock called "continue" to continued the loop above, $i will be at 1, because the loop-increment and loop-test will have run
            if ($did_breakOrContinue -and $i -eq 0) {
                $did_break = $true
            }
        }
    }
    End {
    }
}

1..100 | Foreach-ObjectImproved {
    if ($_ -eq 20) {
        break
    }
    if ($_ % 15 -eq 0) {
        "fizzbuzz"
        continue
    }
    if ($_ % 3 -eq 0) {
        "fizz"
        continue
    }
    if ($_ % 5 -eq 0) {
        "buzz"
        continue
    }
    $_
}

Foreach-ObjectImproved in the code above uses some trickery to detect when the provided scriptblock does break and continue, and then acts accordingly, again, turning something that actually shouldn’t work into something that does. Makes me wonder why Microsoft didn’t implement it this way to start with. Or at least something like it.

(I take no responsibility for Foreach-ObjectImproved, it’s an awful hack and is probably subtly broken in some way that’s not immediately obvious to me. If it breaks, you get to keep both pieces.)

We run a Zimbra mailserver at our company. We’re trying as hard as we can not to run it, because there’s really no real value in running your own Zimbra mailserver when most of your users want Outlook anyway, and cloud email services like Office 365 are so affordable. But we do it anyway, for Business Reasons.

Anyway, that’s not really the point of this post, so I won’t dwell on that. The important part is that the server is an Ubuntu 14.04 LTS server, that has been upgraded over the years, and probably started out at Ubuntu 10.04 LTS or similar. Virtual machines have a tendency to outlive the hardware they’re on.

We have a recurring issue with the VM. Automatic OS updates are enabled, including installing new kernels. However, automatic updates on Ubuntu do not seem to remove older kernel versions for us, so the /boot partition, where the kernel among other things are stored, will often fill relatively rapidly. When it does full, apt becomes sad and broken, and we have to clean up the mess manually. In order to mitigate this, we’d like to just increase the size of /boot on that box. On this box, /boot was on the primary partition /dev/sda1, and is just a dismal 228 MB in usable size.

As an aside, I know that historically it used to be important to locate your kernel near the start of the drive, because LILO (which historically was the bootloader of choice for Linux) couldn’t read past the first GB or so of the drive. I really doubt that’s still an actual issue with Grub, but history lives on I guess. Bootloaders are still a bit of an arcane magic to me. Anyway, I really want to change the existing system as little as possible, because it’s a production machine.

Ideally, what I want to do is to resize the /boot partition to make it larger. Before even looking, I knew that was going to be a pain, because of the partition that was inevitably directly following it. So I looked a little closer:

# df -h
Filesystem               Size  Used Avail Use% Mounted on
udev                     3.9G   12K  3.9G   1% /dev
tmpfs                    799M  596K  798M   1% /run
/dev/mapper/zimbra-root  2.0T  625G  1.3T  34% /
none                     4.0K     0  4.0K   0% /sys/fs/cgroup
none                     5.0M     0  5.0M   0% /run/lock
none                     3.9G     0  3.9G   0% /run/shm
none                     100M     0  100M   0% /run/user
/dev/sda1                228M  101M  116M  47% /boot
/dev/sdb1                9.8G  1.6G  7.7G  17% /opt/zimbra/redolog
/dev/sdc1               1008G  680G  278G  72% /opt/zimbra/backup
# pvs
  PV         VG     Fmt  Attr PSize PFree
  /dev/sda5  zimbra lvm2 a--  1.99t 24.38g
# lvs
  LV     VG     Attr      LSize  Pool Origin Data%  Move Log Copy%  Convert
  root   zimbra -wi-ao---  1.95t
  swap_1 zimbra -wi-ao--- 16.00g
# fdisk -l /dev/sda

Disk /dev/sda: 2190.4 GB, 2190433320960 bytes
255 heads, 63 sectors/track, 266305 cylinders, total 4278190080 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006250a

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048      499711      248832   83  Linux
/dev/sda2          499712  4278190079  2138845184    5  Extended
/dev/sda5          501760  4278190079  2138844160   83  Linux
#

Aha! There is hope after all! The /dev/sda1 partition, which I’d like to make bigger, is actually followed by an extended partition, which contains only one logical partition, /dev/sda5, which is a physical volume for an LVM volume group.

My plan becomes:

1. Hook a temporary scratch disk large enough to temporarilly hold the contents of the LVM vg “zimbra”.
2. Initialize the scratch disk and add it to the “zimbra” volume group.
3. Evacuate the /dev/sda5 physical volume onto the scratch disk.
4. Remove /dev/sda5 from the volume group and remove the /dev/sda5 and /dev/sda2 partitions.
5. Increase the size of the /dev/sda physical disk in order to accommodate the new expected load of data.
6. Change the end of the /dev/sda1 partition in order to increase its size to a few gigabytes or so.
7. Perform an online filesystem resize of the /boot partition.
8. Move the data back off the scratch disk.
9. Scrap the scratch disk.
10. Make a storage clone of the running server, and make sure it should boot properly.
11. Reboot the production server just to make real sure.

Note that there’s no no step in which the server becomes unavailable for a lengthy during a data move. This looks like a good plan on paper!

The final reboot step is just in there for extra insurance, I’d rather have a reboot fail when I’m there to fix it, outside of regular working hours, than having it fail in uncontrolled circumstances.

Let’s see if we can execute the plan.

Originally, my plan involved shrinking the /dev/zimbra/root volume to about 1 TB, down from 2 TB, to speed up any moves, and negate having to resize the virtual disk. However, I was disappointed to find out that online filesystem shrinking is not possible with ext4.

Because this involves a critical production workload, which you really don’t want to have to roll back from backup (lost mail and all that), I wanted to test all the steps first on a production environment, so the first step was for me to take a clone of the virtual machine. In my environment, we use Nimble Storage, and we had the Zimbra stuff on one LUN, so it was exceedingly easy to just create a snapshot clone of the VMFS volume the Zimbra VM lives on, and then add the cloned VM to the VMware inventory.

After removing the virtual network card (important, because I really don’t want the lab clone touching any real stuff), I powered up the VM. It took a few minutes longer than usual, because of the system and Zimbra itself grumbling about missing networking, but I really didn’t care about that. It was to be expected, but eventually my clone booted, ready for me to perform my experiments on.

---

Step 1: Hook a temporary scratch disk large enough to temporarilly hold the contents of the LVM vg “zimbra”.

First order of business was to hook up a scratch disk. Because I don’t want scratch data blowing up a thin provisioned datastore, I created a new LUN on our Nimble Storage, 3 TB in size, for the purposes of holding scratch data. I then created a 2 TB virtual disk that I hooked to the lab VM.

I make a mental note that when I’m doing this for real, I should add the scratch LUN to the Nimble Storage volume collection that’s periodicaly taking storage snapshots of our mailserver, that way we have consistent snapshots in case something awful would happen right in the middle of our move.

Looking at the console, it seems the scratch disk received the device name /dev/sdd.

Step 2: Initialize the scratch disk and add it to the “zimbra” volume group.

This part was easy.

# pvcreate /dev/sdd
# vgextend zimbra /dev/sdd

Done!

Step 3: Evacuate the /dev/sda5 physical volume onto the scratch disk

This part was easy too, once you got past the slightly too simple syntax.

# pvmove /dev/sda5

This is all I needed to type. The parameter the disk that I want to evacuate (to any other drive(s) in the volume group, in my case just /dev/sdd), which perfectly aligns with my use case.

The only hard part was the time taken. This took about 8 hours to run on our setup, so I left this run until the next workday. I figured that’s entirely sensible to kick off at like 10 PM to run until 6 AM, during the “quiet hours” of the mail system.

Step 4: Remove /dev/sda5 from the volume group and remove the /dev/sda5 and /dev/sda2 partitions.

More easymode!

# vgreduce zimbra /dev/sda5
# fdisk /dev/sda

Command (m for help): d
Partition number (1-5): 5

Command (m for help): d
Partition number (1-5): 2

Command (m for help): w

I did get an error message “Re-reading the partition table failed with error 16: Device or resource busy”. The error message suggested to run partprobe, so I did. I’m not sure if it actually did anything, but if anything it shouldn’t do any harm.

# partprobe /dev/sda

Step 5: Increase the size of the /dev/sda physical disk in order to accommodate the new expected load of data.

This is where I ran into a small issue. The drive was very oddly sized at 1.9921875 TB. I thought I’ll just bump it up to like 2.1 TB. That should be plenty. That’s when VMware decided to give me this:

Hot-extend was invoked with size (4509715660 sectors) >= 2TB. Hot-extend beyond or equal to 2TB is not supported.
The disk extend operation failed: msg.disklib.INVAL

Fortunately, the message was quite clear. All right, so I can’t extend it to 2.1 TB. How much can I extend it, what exactly is 1.9921875 TB? Flipping over to “GB” it informs me that it’s exactly 2040 GB, which seems like an arbitrary number, but at least it’s below 2 TB, and there’s just enough space to do what I want to do. Thank God that whoever was before me working on this had the foresight to leave a few GBs of possible expansion. It likely was actually myself, but I can’t recall actually thinking about it, I probably just felt 2040 was a rounder number than 2047.

Anyway, I expand the drive to 2047 GB in size using vSphere Client and it happily complies.

After that, I issue the magic incantation:

echo 1 > /sys/class/block/sda/device/rescan

This is to that Linux can rescan the physical drive to recognise the extra 7 GB of space.

Step 6: Change the end of the /dev/sda1 partition in order to increase its size to a few gigabytes or so.

Because I was only able to grow the disk by 7 GB, this also limits the maximum size of the boot partition to approximately 7 GB + 250 MB (its previous size). If I still want to be able to fit the data back there. I decide to not think too hard about it, and just make the boot partition be 7 GB in size, that should leave some headroom in case things go wrong.

# fdisk -l /dev/sda

Disk /dev/sda: 2197.9 GB, 2197949513728 bytes
27 heads, 59 sectors/track, 2694833 cylinders, total 4292870144 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006250a

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048      499711      248832   83  Linux

As we can see, the drive has increased from ~2190 GB to ~2197 GB. (Note: These are power-of-10 gigabytes, not power-of-2 ones, which explains the apparent discrepancy from the other numbers.)

So, noting that the partition has to start at 2048, we create the new partition and set it bootable:

# fdisk /dev/sda

Command (m for help): d
Selected partition 1

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-4292870143, default 2048): 2048
Last sector, +sectors or +size{K,M,G} (2048-4292870143, default 4292870143): +7G

Command (m for help): a
Partition number (1-4): 1

Command (m for help): w
The partition table has been altered!

Caling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

Again I decide to throw in a partprobe. Can’t hurt.

# partprobe /dev/sda

Step 7: Perform an online filesystem resize of the /boot partition.

This was painless, I just issued:

# resize2fs /dev/sda1

And it just worked. Or so I thought! (Cue foreboding ominous music!)

Step 8: Move the data back off the scratch disk

OK, this is actually several steps. Create the new data partition, initialize it as a LVM physical volume, add it into the LVM volume group, and pvmove the data off the scratch disk. I decided to keep the layout the same, i.e. an extended volume covering the rest of the drive, and a logical volume inhabiting the full extent of that extended volume.

# fdisk /dev/sda

Command (m for help): n
Partition type:
   p   primary (1 primary, 0 extended, 3 free)
   e   extended
Select (default p): e
Partition number (1-4, default 2): 2
First sector (14682112-4292870143, default 14682112): 
Using default value of 14682112
Last sector, +sectors or +size{K,M,G} (14682112-4292870143, default 4292870143): 
Using default value of 4292870143

Command (m for help): n
Partition type:
   p   primary (1 primary, 1 extended, 2 free)
   l   logical (numbered from 5)
Select (default p): l
Adding logical partition 5
First sector (14684160-4292870143, default 14684160): 
Using default value of 14684160
Last sector, +sectors or +size{K,M,G} (14684160-4292870143, default 4292870143): 
Using default value of 4292870143

Command (m for help): w
The partition table has been altered!

Caling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

With the partition created, it’s just:

# partprobe /dev/sda
# pvcreate /dev/sda5
# vgextend zimbra /dev/sda5
# pvmove /dev/sdd

And then, wait another 8 hours.

Step 9: Scrap the scratch disk.

First, just remove the PV from the volume group:

# vgreduce zimbra /dev/sdd

Then, I just hot-removed the VMDK from the VM.

Step 10: Testing

This time around, I’m doing this on the test box, which has no production data flowing through it, and also no network connectivity. So I took a VMware snapshot (including memory) of the state of the machine at this point. The idea is that I can always revert the snapshot and end with the machine booted but not bootable, so that if something goes wrong, I can go “back in time” to fix it.

Expecting the box to reboot successfully, I just rebooted, and I was greeted with failure:

error: unknown filesystem.
Entering rescue mode...

Long story short. I try a lot of things at this point, including going back and setting the bootable flag on /dev/sda1 (which I had forgot to set previously, but which was irrelevant), trying different partition sizes etc. Nothing seemed to work. This is when I stumbled upon Debian bug 766799: On-line resizing of root filesystem with resize2fs makes system unbootable; grub says: unknown filesystem.

Long story short, in my particular scenario, resize2fs made a valid ext2 filesystem, using a feature that grub is not able to recognize. I decide to fix this problem by reverting the snapshot and simply re-creating the /boot filesystem (copying its contents away to a different folder), using the following steps:

# cp -a /boot /boot2
# umount /boot
# mkfs.ext2 /dev/sda1
# mount -a
mount: special device UUID=...... does not exist

At this point, I edited /etc/fstab to substitute /dev/sda1 for the UUID. I could also have replaced the UUID, but I was running with a console with no convenient copy-paste support.

# mount -a
# cp -a /boot2/. /boot
# rm -rf /boot2
# update-grub
# grub-install /dev/sda

At this point, I take another snapshot with memory, and rebooted. And, lo and behold, it rebooted fine!

At this point I was feeling smug and satisfied that I had caught a problem with my procedure before actually performing it, and I pat myself on the back on my good thinking of me to run this on a clone before breaking my production box!

Step 11: Doing it for real

At this point, I feel comfortable with doing this for real.

First, I go and nuke the VM and  the storage snapshot clone that I was using for testing. I won’t be needing that any more, after all. That also gets rid of the scratch VMDK inside the scratch datastore, leaving it empty.

Currently, the Zimbra volume was protected as a standalone volume in or Nimble Storage array. In order to maintain our data protection posture, I changed the Zimbra volume to be protected as a part of a volume collection instead of as a standalone volume, and then proceeded to add the scratch volume to that volume collection.

Having configured that, I go back, and repeat all the steps from earlier in the post.

When the time comes to do the first pvmove, instead of invoking it immediately, I delay it to run at 22.00, like this:

# at 22:00 today
warning: commands will be executed using /bin/sh
at> pvmove -b /dev/sda5
at> <EOT> (I pressed Ctrl+D)
job 2 at Thu Apr 14 22:00:00 2016
#

I added the -b flag in order for pvmove to run in the background rather than in the foreground, since I’m starting it off an at job.

Returning to work the next morning, I breeze through steps up until the online resize of the filesystem. Because I already know that won’t work, I instead copy the contents of /boot away to the root filesystem, blow away the old partition, fix up /etc/fstab, copy the stuff back, and then run grub-update and install-grub, as outlined earlier. Although, since I’m now working with a terminal with copy-paste capabilties, instead of replacing the uuid with the device name in /etc/fstab, I simply update the UUID to the correct one.

Once I reach the step to do the second pvmove, again, I use at to delay it to 22:00, as above.

At this point, I decide that it’s a good idea to take a snapshot clone of the Zimbra volume collection and mount up a clone of the VM to ensure that booting works correctly. The location of the actual data on LVM would seem to be immaterial. After cloning up the two datastores, and adding the clone VMX to inventory, I have to adjust the VM to point the scratch disk to the cloned datastore, as well as remove the network card. After starting the clone, it boots just fine, which confirms that I did everything right.

So, all that remains is copying the data back off the scratch drive, and scrapping the scratch drive, but that’ll have to wait overnight.

After returning the next morning, I see all the data having been moved. So, as a final step:

# vgreduce zimbra /dev/sdd

Then, I removed the VMDK from the VM.

I decided to wait until the next storage snapshot rolls around on the hour before finally, killing the scratch LUN that it lived on, in the exceedingly unlikely event I’d need something off it.

All that remains now is rebooting the server to make sure it boots properly for real. I’m resonably confident it will, given all my precautions, but you never really know until you actually do it. That’ll have to wait for the maintenance window conveniently scheduled tonight.

After an embarrassing typo, the server was ultimately rebooted an hour prior to the actual service window, nevertheless, the server booted properly. Yay!

At the end, what really bit me in the ass wasn’t what I tested, it was what I didn’t test. I had not considered that the virtual machine had been configured to only back up certain disks of the mail server, which meant that the backups from the time the scratch disk was in use were not complete or usable. Fortunately, Zimbra keeps its own backups, which mitigated this. This actually turned out to be a boon, becuase this way only one data move had to be backed up into an incremental file, cutting in half the backup load, which is important because of our very slow off-site backup replication, that will take a few days to catch up, because of this.

Another thing that bit me in the ass is that I did not consider that the backup job would be running at the same time as the data move, meaning a VMware snapshot was present on disk during the time of the data move, which in general is a big no-no. In this case, it doesn’t actually seem to have affected anything, though.

I guess the moral of the story is to always consider what the backup system might do during an operation like this. I considered the Nimble Storage snapshots, but not backups, which was an oversight.

At my company, we sometimes take on new customers where we want to replace their existing firewalling, in order to bring them into our standard operating environment, which involves fitting into a certain IP addressing plan (making it possible for us to deliver services from our datacenter without IP conflicts.

This usually involves changing IP addresses at the new company, as well as changing out their firewalling. Sometimes, an on/off migration isn’t practical, because there’s a lot of systems that have to be renumbered, while maintaining functionality.

For example, in a current ongoing migration project, the customer has laptops that connect through a “road warrior” VPN to the existing box, and we want to be able to provide both methods for connecting during a transition period, to eliminate any disruption.

What I’m describing in this post is a method I like to use to acheive what I like to call a staged transition network migration. I’m not going to go in deep on the specifics on how to operate a specific product, because it really is impossible to do, because every vendor does things differently. This method general enough that it should work no matter what equipment you’re moving from or to. I can’t really take credit for this method, though, because even though I came up with it independently, it’s basic and simple enough that I’m sure many other people have independently come up with and used the exact same technique in the past.

It looks like this at the Layer 3 level:

                    +------------+
                    | ISP Router |
                    +-----+------+
                          | .1
..........................:.........................
: Outside internet-accessible network 192.0.2.0/29 :
:..................................................:
      | .2        .................          | .3
+-----+------+    . Transition    .    +-----+-----+
| Old creaky |____: interconnect  :____| New shiny |
| firewall   | .2 : 10.99.14.0/24 : .1 | firewall  |
+------+-----+    :...............:    +------+----+
       | .1                                   | .1
.......:.........                  ...........:.....
:    Old LAN    :                  :    New LAN    :
: 172.16.0.0/24 :                  : 10.99.14.0/24 :
:...............:                  :...............:

In the diagram above, I’m only showing one LAN attached to the old creaky firewall and the new shiny firewall. This is to simplify the diagram, the general design works just fine for multiple networks on either side. These extra networks might be DMZ’s, or VPN client networks. I’m also not showing the VPN to the datacenter.

Physically, this operation consists of putting a layer 2 switch in-line between the ISP CPE (Internet Service Provider Customer Premise Equipment) and the old firewall. This causes a short disruption to the internet connection (a matter of seconds if you work quickly). For this to work, you need spare IP addresses with their ISP. It may also be a completely separate internet connection if you’re transitioning that as well.

A layer 2 connection is brought up between the firewalls. Typically, this will be an Ethernet cable (cross-over if you don’t have Auto-MDIX) between a free interface on the old firewall and an interface on the new firewall, but it might also be a VLAN.

I will typically create a new VLAN for the “New LAN” in their existing switches, and hook the new firewall’s LAN port up to that.

After that, you just add routes on each firewall, so that they know to send traffic that is destined for the other firewall’s lan through the other firewall.

After that, you just have to make sure the firewall policies on both sides will permit traffic to pass, and just like that, you have two networks that can talk to each other.

Now, it’s just a matter of moving boxes from one VLAN to the other. That’s easy to do by just logging into the switches and changing the VLANs on a port-by-port basis. For devices that are DHCP-configured, I like to disable the switchport and re-enable it to trigger a DHCP renew on the end device. Otherwise, it might not realise that it’s on a new VLAN and linger with the old IP.

Then, once you don’t have anything still connecting through the old networks, you can just take it down with no special ceremony. You might remove the in-line switch that you installed to enable both firewalls to be connected at the same time, to eliminate a potential point of failure, if it is no longer neccessary.

Backup is a royal pain in the net. There’s a lot of backup software out there. Some good, some not so good.

Windows Server Backup (or ntbackup) is marginally useful and dangerous at worst. I’ve seen it set up at some small shops, mostly because it’s free, but I have two main pain points with it.

First of all, Windows Server Backup will not back up to drives with 4 KiB sectors. We had a customer who was backing up to a USB drive that failed. We brought in a new USB drive, that had 4 KiB sectors, and it would not back up. Googling found lots of angry people but no solutions. That, and the fact that it’s nigh on impossible to find out if a drive has 4 KiB or 512 B sectors before buying, makes this an ongoing problem.

Let me repeat that, in case that happened to glide past you. The backup destination drive, i.e. where the backup images themselves are stored, i.e. not the drive being backed up, has to have 512 B sectors. This is insane. Why would software ever need to care so deeply about where the files are being stored? There’s probably a limitation saying that the drives to back up also have to be 512 B, but don’t quote me on that. I haven’t tried.

Second, Windows Server Backup has no sensible way of telling you when backup fails. This is insane. You setup a backup solution, it seems to work, then something or other breaks, and it fails, writing an event to an event log that nobody reads often enough to notice. This makes it dangerous, unless you have as a procedure to read your event logs daily at best, or at the very least on a regular basis (not just reading them in the case of a problem). At worst, the problem can persist for months or even years until your main drives actually fail, and you’re caught with your pants down with no recent backup.

Microsoft’s strategy to deal with this piece of suck is to put it out to pasture, pretend it doesn’t exist and not fixing any problems.

Now, I’m told that you can set up alerting on errors with Windows Server Backup in different ways – one way I’ve heard is using Task Scheduler to trigger sending an e-mail on different Event IDs using a script.

The problem is, I don’t trust myself not to screw this up. You basically have to know in advance what log events to be looking for, i.e. enumerate all possible bad conditions, before sending out logs. This is not something individual sysadmins should need to do, that’s the job of whoever makes the software. Besides, I’m not likely to do that kind of work for software that won’t even back up to (i.e. store backup files on!) 4 KiB drives, and that Microsoft considers dead.

So… whenever I see it, I try to get it ripped out and replaced. With what? Well… that’s a good question…

So, yesterday, I had the requirement to shrink a partition. The partition was spanning four dynamic disks, with a total size of 7.5 TiB.

So, using Disk Management, I just go into the GUI, right-click this monster of a partition, click shrink, wait a couple of minutes for it to analyze the free space, and then to tell it to shave of a little north of 4 TB. The goal was to get rid of the two last dynamic disks to reclaim disk space. I fully expected this to take its sweet time. After all, moving data takes time, especially up to 4 TB of it. For the shrink to work, it basically has to pick up all the data past the end of the new partition, and jam it in at the start of the drive.

I was expecting like… a progress indicator which you might expect on any long-running operation? Nope. All I got was a Windows 3.11-style hourglass mouse cursor whenever my mouse is inside Disk Management. Totally useful.

So I leave it overnight, and it’s still running. Curious to see what the hell’s going on, I look at the Resource Monitor. Under Disk, I see that svchost.exe (defragsvc) is reading and writing to my disk at 20 MB/s. Not exactly high speed, but meh. Back-of-the-envelope calculation tells me that 4 TB at 20 MB/s would take 60 hours. That’s fine, it can run over the weekend (I’m not too fussy about how long this takes even though it’s dog slow), but I still wanted to find out its progress. Because I like micromanaging my computers. Like a project manager, I demand percentage-based progress reports!

The fact that it’s defragsvc doing the work is an important clue (and it makes sense too, after all, it is a kind of a full-disk defrag that has to happen). So, I go on the Googles (defragsvc progress), and I click on a random vaguely promising link and end up at the Windows Seven Forums, on a thread labeled “defrag display”. I scroll past most of the stuff that’s not relevant to my case ut then I found this one post listing a few interesting switches to defrag.exe – the command-line interface to defrag. The one that caught my eye was not the one in bold, but rather these two:

/T – Track an operation already in progress on the specified volume.
/U – Print the progress of the operation on the screen.

So… on a whim, I open up a shell and type in “defrag d: /t”. Of course, whoever installed the server made it in Swedish. I have nothing against localized operating systems, in fact I make it a point to make my own PC run in Swedish, but servers need to be installed in English, but I digress. (I’ll probably write something on that.) Anyway, this is the output I got:

PS C:\Users\Administratör> defrag d: /t
Microsoft Diskdefragmenteraren
Copyright (c) 2007 Microsoft Corp.

Spårar åtgärd på Data (D:)...

That says “Tracking operation on Data (D:)”. But there’s no actual other information. I wonder about “what’s the point of tracking an operation if you don’t see any progress?” so I Ctrl+C out of that and instead try:

PS C:\Users\Administratör> defrag d: /u
Microsoft Diskdefragmenteraren
Copyright (c) 2007 Microsoft Corp.

Anropar defragmentering på Data (D:)...

 Analys: 100 % färdigt.

Rapport före defragmentering:

 Volyminformation:
 Volymens storlek = 7,49 TB
 Ledigt utrymme = 4,94 TB
 Totalt fragmenterat utrymme = 5 %
 Storlek på största lediga utrymmet = 599,77 GB

 Obs! Filfragment som är större än 64 MB inkluderas inte i fragmenteringsstatistiken.

Utför pass 1:
 Defragmentering: 0 % färdigt...

So… it actually started a new defragment. I also got a pop-up message from Windows (probably from Disk Management) saying my shrink operation was aborted. Balls. It seems that I told it “defrag drive D, and show me the progress as you go”. Not what I wanted. I Ctrl+C’ed out of that as well. I would have been more annoyed at having the defrag aborted half-way, but the fact is that the subsequent defrag run will just have less to do anyway, and I’m not really in a rush.

So, then I thought, maybe you need both switches? Would make sense, in a perverse Microsoft way. Track the operation and show me the progress.

So, I click through starting the shrink process again, and then I try this:

PS C:\Users\Administratör> defrag d: /t /u
Microsoft Diskdefragmenteraren
Copyright (c) 2007 Microsoft Corp.

Spårar åtgärd på Data (D:)...

 Analys: 100 % färdigt.

Rapport före defragmentering:

 Volyminformation:
 Volymens storlek = 7,49 TB
 Ledigt utrymme = 4,94 TB
 Totalt fragmenterat utrymme = 5 %
 Storlek på största lediga utrymmet = 599,77 GB

 Obs! Filfragment som är större än 64 MB inkluderas inte i fragmenteringsstatistiken.
 Krymp: 53 % färdigt...

That looks promising. “Krymp” is means “shrink” in Swedish. So it’s actually showing me progress of the exising operation. Yay!

I found it interesting that it started out at 53% and not at 0% – maybe it recognised it had been aborted? Or maybe it calculates the progress from the “worst case possible” shrink operation…?

Would be nice with a decimal point or two more, so you can actually stare at it and see it moving. Or maybe a bogus ETA? Oh well, it’s a lot better better than just an hourglass. In the end, the best way for me to actually figure out how long this is going to take, is to wait a few hours, see how many percentage points it’s done in that time, and extrapolate.

So, to recap, if you have a current long-running shrink operation happening that you started from Disk Management, if you want to know what’s going on, pull up a shell, and type in:

defrag X: /t /u

Substituting the drive letter you’re shrinking for X, of course.

As a final word of warning: Do not Ctrl+C out of the progress display! If you do, it’ll actually abort the defrag operation! Which actually makes no sense, since you’re just tracking an ongoing operation… but this is what I’ve observed.

Update: A few hours after originally posting this, I decided to check on it, and see how it was doing. It was still stuck at 53% complete. I killed it and restarted it, and it came to 53% done after a minute or so, jumping there directly from 0%. It may well be that the progress indicator simply does not work for whatever reason, and that this entire post is useless.

Update 2: I discovered the same progress information can be found by opening the Disk Defragmenter UI.

Update 3: One day later, we’re at 54%. I think however that this is due to a very large inbound file transfer (on the order of 1 TB) that’s been running for over a day now to this server, at the same time as trying to shrink. So the progress is moving after all. I expect it to start moving a lot faster as soon as that large file transfer is finished.

All the hallmarks of net pain are right there in the title.

• P2V.
• Microsoft Windows 2003 Small Business Server
• Using CloneZilla.
• To Hyper-V.

First of all, Windows Server 2003 Small Business Server is awful. It’s old, it’s unsupported, and the friendly features work against you more than they work with you. Not to mention that as a final fuck you, it decided that it was not permitted license-wise to do a domain trust to a new forest, further complicating our migration. But that’s another blog post whatsoever.

Second, why am I using CloneZilla instead of some more supported tool? Well… there doesn’t actually seem to be any good P2V solutions to Hyper-V. Microsoft has a P2V tool that’s part of System Center Virtual Machine Manager, which we’re not going to install for a single-hypervisor node. So, that means we’re on our own from Microsoft. I don’t even think their official P2V even supports Windows Server 2003 anyway, but I digress.

The other go-to method for a simple P2V into Hyper-V is using the awesome tool called Disk2vhd. I’ve used this in the past to great success, but I decided not to use this method this time, because the machine I’m cloning from is a domain controller and a mail server. It’s not a good idea to clone these online, because the clone you get will always be stale, and you get “time warp” effects including lost e-mail that might have been delievered while the clone was in progress.

So, I elect to use CloneZilla. I plan on using it in disk_to_remote_disk mode, as documented in this Wiki artickle from Ogala Lakota College. Last edited in December 2009. So the article is over six years out of date already. I feel the pain already…

My plan is to run the CloneZilla live CD on the Hyper-V box and use that to receive the clone, after booting the physical box with a burned CD of the ISO file of CloneZilla. No sense risking flaky USB booting on a machine of that vintage. My fall-back plan is to isolate the server from the network, and use Disk2vhd after all, which should in theory be fine as long as the server doesn’t talk to the network.

I download the ISO and for a moment I have to double check that the old creaky hardware actually supports 64-bit OS’es. Turns out it does. Armed with Plan A and half of a Plan B, I pack my backpack for the evening.

---

Arrived at the customer site, armed with the CD. I boot the machine with it. Notice the RAID array at the customer is degraded with both disks healthy. Maybe that had been causing some issues? Never mind, that’s exactly why I wanted to P2V.

After some ominous read errors on the CD drive, CloneZilla live booted without a hitch. Actually, the instructions seemed to be entirely up to date despite being written in 2009. Yay for stable software.

At some point, I’m presented with a choice between two hard drives. Hmm. Seems CloneZilla isn’t fooled by the server’s FakeRAID and presents the two drives seperately. Good thing it’s a RAID-1. Also, the array was degraded anyway, so I knew one of the two drives had to be stale.

I pressed Alt+F2, entering into the command shell, become root, and examine the situation using fdisk. Both drives have two NTFS partitions. (fdisk /dev/sda , then p to print the partition table, then q to exit without making changes)

I decide to look a little deeper to see which one’s recent and which one is stale. I mount the sda1 and sdb1 partitions read-only using these commands:

cd /mnt
mkdir a b
mount -t ntfs -o ro /dev/sda1 /mnt/a
mount -t ntfs -o ro /dev/sdb1 /mnt/b

After a bit of poking around and looking at file dates, it’s apparent that the copy on /dev/sdb is up-to-date. The other one is quite stale.

At that point I unmount the two partitions:

umount a
umount b

And return back to the menu’s using Alt+F1. Certain of my choice, I choose to clone from /dev/sdb

After following through, it gives me a few commands I need to type into the VM where I had booted CloneZilla before. The source machine is saying “Waiting for the target machine to connect…”

I enter the commands on the target machine, and a progress window shows up on the target machine! Success! Although there is no indication that there’s any file copy happening on the source machine… oh well. It’s definitely copying.

It asked a few questions regarding whether to copy over the boot loader etc, I just said yes.

I leave and come back to a progress bar at 90%. I watch it go to 100%, and then I realised I’d been trolled. The progress bar is volume by volume. Now, I still have to wait for the second volume to finish copying… all the while standing in the hot and untidy server closet at the customer, with nowhere to sit. I text my wife I expect to leave in about an hour 45 minutes – around 20.00 – after all, there’s another 22 minutes left on the file copy, and I’ll probably need just north of an hour to actually get the server to work right. Faith is high.

At this point, I’m waiting patiently, expecting the VM to come back up after reboot, probably screaming bloody murder about drivers, but in general being in a fixable state. I hope. 5 more minutes. I wait.

I write a hand-written note. “This machine – SRV03 – HP Proliant ML110 – must NOT BE STARTED”. Best to cover all the bases since the old server is not labeled with a hostname. I plan on unplugging the power from the server as soon as the copy has finished. Better safe than sorry with two almost-same but different SBS servers duking it out on the network in a Small Business Deathmatch. I duct tape the label over the power button. Oooh, copy complete. I restart the virtual machine after disconnecting the virtual CD-ROM, and I’m met by – nothing. A blinking cursor on a black screen.

At this point, I decide to spend a few minutes to find a way to sit down, hook up my laptop and get comfortable. This was going to take some effort.

After a few seconds of Googling (windows 2003 sbs hyper-v black screen cursor), I find a Technet blog article titled VM does not Boot Following P2V or Disk2VHD

Looks like I’m going to need to get Windows Server 2003 boot media. No problem. I grab an ISO off that off the Volume Licensing Service Center. Or so I thought. Microsoft no longer has ISO’s of that on VLSC. Meh. Figures though. I should have been better prepared and got an ISO of that onto the virtualization host just in case. I’m not surprised Microsoft has pulled the ISO’s, the product is no longer under support after all.

I VPN into our datacenter on a quest to locate a Windows Server 2003 ISO. We’re bound to have one laying around. For the first time ever I find the web-based datastore browser to actually be useful. Rooting around in a musty old folder named “From old datastore” I find a copy of Windows Server 2003 R2 32-bit. I’m pretty sure the server was running 32-bit to start, since I saw that it had PAE enabled. It’s not SBS, but for a bootup fix it shouldn’t matter.

I download the files, and then copy them up to the new shiny Hyper-V box. I spend a few minutes admiring progress bars, and mentally berating myself for not thinking of this ahead of time. Mental note: Always make sure there’s a copy of the install ISO for the P2V:ed or V2V:ed operating system ahead of the actual V2V. Better have it copied ahead of time instead of doing it over a VPN in an uncomfortably hot server closet.

I insert the virtual disc into the virtual DVD drive and reboot, and I see the windows installer. I get nostalgic for a moment as the old text-based pre-installer shows up.

I press R for repair, select my keyboard, select the windows install, type in the admin password, and then enter the magic incantations, and confirming along the way:

fixmbr
fixboot
bootcfg /rebuild
exit

I get up two possible boot options after rebooting, Windows Server 2003 for Small Business Server, and my new option I labeled “P2V”. I decide to choose the old option to see if it works. It’s not clean to have two options, so I plan on manually editing C:\boot.ini to clean it up, once I’ve established which one works. Then, I see the old familiar Windows Server 2003 boot screen. I wait.

The GUI loads, preparing network connections. I realise I’ve forgotten to note down the old IP address. No matter, I log on to their old firewall to look at the DHCP scope to find the DNS server, that’ll be it. Stupid. Should have known that setting would get reset, I just did another V2V a few hours ago.

It reaches the login screen, running at 640×480 resolution with no mouse support. Yay. Fortunately, I’m pretty good with keyboard navigation in windows. I insert the Hyper-V integration services disc, log in and make it the first order of business to install the integration services.

Windows wants to activate because of major hardware changes. I tell it to do it later, since it’s not likely to have working networking. At this point I’m ready to declare that I will be successful, so I shut off the source machine (which was still running CloneZilla), and unplug it from the power. I decide to also unplug it from the network. You can never be too safe.

The Hyper-V integration services finish installing, and it asks me to reboot. And the mouse actually works \o/ Faith is quite high. I reboot the server again.

The old crusty SBS takes quite some time to reboot, but it gets there in the end. That’s all the matters. I log in, and make sure to set the correct static IP. Suddenly, Microsoft gets all agressive on me and wants to activate NOW NOW NOW or it’ll shut down. I decide to let it, it should be able to get out to the Internet through a DHCP address. Activation passes without a hitch. Good, last thing I wanted was activation issues.

I investigate the VM config. Really? Only one CPU core? The old physical server had a dual-core Xeon (oooh!), at least it should have two cores… I plan on shutting down and adding another processor core after changing the IP address.

Wait a moment, where’s my explorer? After activating, all I got was a blue desktop background screen (not to be confused with a BSOD). I wait for a few minutes, still nothing. I hit Ctrl+Alt+Del, bring up a task manager, nothing’s taking the CPU. I decide to try to kill and re-launch the explorer manually. Nothing. Hmm. I launch cmd. That works. I decide to log out and then log back in. Still nothing. I decide to try a shut down. Might as well add that second CPU core now. Remembering from the back of my head that going between a single-processor and a multi-processor OS install can screw with things. Not sure if that’s applicable here.

I look at my watch. Half an hour until I estimated I’d get out of here. Looking a little unlikely but not impossible.

Realize after rebooting that the new boot config was the default when booting. Hmm. Maybe that’s a relevant difference? Remind myself again to tweak boot.ini once I get the IP settings right.

Phew. After a reboot, explorer pops up just like it should after reboot. Might have been either a bug with the activation, or with the wrong number of CPU’s, or the boot.ini. Don’t know, don’t care right now. 🙂

When changing the IP, I get warned about the same IP being present on another network adapter. I ignore that for now, making a note to remove the hidden network adapters.

I log on to the server through RDP rather than through the console. Not being stuck at 640×480 is a liberating experience.

I adjust the boot.ini, removing the redundant section from the boot configuration. I find the following in the config:

[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="P2V" 
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003 for Small Business Server" /noexecute=optout /fastdetect

This tells me it couldn’t possibly have been the two different boot options, since they both just do the same thing. I remove the line with P2V in it anyway. Boot.ini wouldn’t save in notepad, until I did this in CMD:

attrib -s -h -r c:\boot.ini

Then I saved, and I did:

attrib +s +h +r c:\boot.ini

Another reboot, to make sure everything seems to stick. I decide to shut it down using the integrations services to make sure that works right. It does.

I check my watch. 19.45. Looks like I’ll be able to leave when I expected after all! Or at least not too far off. I start packing my stuff, and type in the details in my phone to order an Uber home.

The server reboots, is RDP:able, I check the event log, nothing unexpected, looks good. Server sure is snappier than on the old hardware. Customer’s probably gonna like that. I shut down my computer and head home.

On the way home, I text the customer to call me in the morning, if anything’s broken. I plan on sleeping unmolested, though.

On-site off-hours work is just a pain in the net. But it could have been worse.

 

Let’s say I have my own hosting platform based on vSphere, where I run most of my customers, as a service provider, on a decent cluster with multiple nodes and shared storage and all that jazz.

Let’s further say that I have customers that have local Hyper-V environments running on single boxes here and there, because those customers have chosen, because Reasons, that they want local hardware. The VMs themselves as well as Hyper-V in my test scenario are all running Windows Server 2012 R2.

But what if that hardware breaks? What if there’s a disaster of some sort that makes their server inoperable and their offices uninhabitable? No fun, we have to wait for the hardware to be repaired/replaced, and then do restores. What if we could instead do the restores to a virtual machine in our environment? After all, we do use Veeam Backup and Replication (which is a nice product by the way), and we might already let them store backup copies in our datacenter.

Well, no such luck, because Veeam can’t restore a Hyper-V VM onto vSphere, nor vice versa. Which kind of makes sense, V2V is a hard problem that Veeam doesn’t seem to want to tackle.

We might consider setting up a Windows Server with Hyper-V or two in our datacenter to deal with these potential use cases, but that’s hardware that would just sit unused most of the time. So, that probably won’t happen here.

But all isn’t neccessarilly lost! What if you could use Veeam to Restore VM Files rather than restore the VM itself? That would basically tell Veeam to dump out the constituent files making up a Hyper-V VM, in this case the configuration file (XML) and the VHDX files, and then use some kind of tool to import the Hyper-V VM into VMware from that intermediate restored format? Let’s see.

The first that comes to mind is VMware vCenter Converter. I have that installed on my vCenter box already for other reasons. It looked quite promising, after all, it advertises itself as being able to convert Hyper-V VM:s, but if you look at it a little closer, it seems like it can only do conversions from actual Hyper-V boxes, not from a set of VM files.

This discouraged me a little, and I looked around for other tools. I found (but didn’t try) Starwind V2V Converter, which I probably would have ended up going with, if it weren’t for the fact that it’s a hassle to basically create a new VM and then attach disks to it. There had to be a better way.

That’s when I thought about VMware vCenter Converter again. It has to talk to a Hyper-V server. It has to be registered there, but it also has to be powered off! Aha!

So… what if I would in fact set up a Windows server with the Hyper-V role? I would not have any intention of actually running any VMs on it, just use it to register VMs being restored off Veeam Backup and Replication, for the purposes of VMWare vCenter Converter then doing a V2V of it onto our real production cluster?

I set out to try it. I decided to use my Veeam Backup server (running Windows Server 2012 R2) as the Hyper-V box. It’s already a physical disk with a ton of Storage Spaces JBOD, and I wouldn’t have to send the data over the network for the initial staging of the VM.

So, I decided to start out by simply adding the Hyper-V role to the box.

It asked me about virtual switches, I just skipped that, since I don’t intend to power on any VMs on the box.

It asked me about setting up Live Migration, I skipped that as well, since we’re not doing that.

It asked me about default storage locations. I decided to leave it at the default for now, and see about adding some scratch storage for receiving the restore. For a simple test just doing it on the C-drive where there was ample space was good enough.

After rebooting, I went back into the Veeam Backup & Replication console, and went to Backup Infrastructure and added a new Microsoft Hyper-V Server – the backup server itself. It had to install a few components and took a few minutes.

After that, I made Veeam Backup & Replication restore the entire VM to the backup server where I just installed the Hyper-V role. After the restore was complete, I looked over at the Hyper-V manager on the backup server. Indeed, the VM was there but powered off. I did not attempt to power it on.

Instead, I went over to my vCenter server, where I have vCenter Converter installed. I might just as well have had it on the backup box, but this is where I already had the software. I pressed “Convert Machine”, selected “Hyper-V Server” as a source type, and pointed the software at my backup server.

Here’s where I ran into a hiccup related to Windows Firewall. The diagnostic logs (within two layers of ZIP files) of vCenter Converter showed that it was attempting to connect on port 9089, like so:

2016-02-23T14:09:29.947+01:00 warning vmware-converter-worker[03124] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x0169b440, h:1700, <TCP ‘0.0.0.0:0’>, <TCP ‘192.0.2.123:9089’>>, e: system:10060(A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond)

As far as I can tell this is what happens: If that port shows “Connection refused” then it would offer to install the proper agent on the remote system. If it just timed out (which is what Windows Firewall will do when somebody connects to a port that doesn’t respond, due to Stealth Mode) then it assumes there are network issues and does not offer to deploy the agent.

Advice I found online said to disable the Windows Firewall, but that’s not how I roll. Instead, I created a firewall exception for TCP port 9089 from the vCenter server to my backup server, and then disabled stealth mode in Windows Firewall. There’s no supported or GUI way to do this, you’ll have to dive into the registry.

Following the instructions from this Server Fault answer, I created the registry key HKLM\Software\Policies\Microsoft\WindowsFirewall\DomainProfile, and a DWORD value under it named DisableStealthMode set to “1”. After that, I restarted the Windows Firewall service. (That, or just save yourself the hassle and install the converter on your Hyper-V host.)

After that I was able to proceed to actually deploy the agent in VMware vCenter Converter. Yet another pain in the net…

I chose to manually uninstall the files later, as I anticipate to perform more conversions in the future, but then:

The version of the selected Microsoft Hyper-V Server is not supported. Select an earlier version of Microsoft Hyper-V Server or another supported source type.

More pain. Looking at the release notes, it turned out that I have an older version of VMware vCenter Converter installed – version 6.0 doesn’t seem to support Windows Server 2012 R2 according to the release notes. Fortunately, version 6.1 does seem to support it. So, I proceeded to upgrade it. I also tested re-enabling Stealth mode on the backup server’s Windows firewall to see if the same bug was still present with a recent version. Looks like this bug is still present as of version 6.1.1 build 3533064, which is the most recent as of posting. Oh, VMware, why don’t you test your crap with Windows Firewall enabled? Disabling Windows Firewall may be a common practice, but it’s hardly best practice.

Anyway, now I repeated my steps, and again, I entered the hostname and the credentials, and entered my credentials. And YES! I see my powered off VM I want to get into vSphere.

So, I choose it, and press next, and here comes more pain.

Permission to perform this operation was denied.

Back into the diagnostic logs. This time, I find the appropriate log within three layers of nested ZIP files. Somebody at VMware must have a real fetish for Russian matryoshka dolls, all that nested unzipping… Fortunately, the cause seemed quite simple this time:

2016-02-23T15:08:20.277+01:00 warning vmware-converter-agent[06692] [Originator@6876 sub=Default] UAC is enabled! using non elevated administrator ‘contoso\pv2b-admin’

So, I restart VMware vCenter Converter Standalone again, and this time, elevated as an administrators. Convert machine, powered off, servername, user name password, deploy agent, select VM, next, and again. Permission denied.

This time, I guess the problem is probably Remote UAC being enabled, since it’s trying to get admin over the network. Yet more registry hacks chipping away at security… incidentally Veeam needs the same to work, so I figure I’d try that, according to the solution I found over at Xyfon solutions.

Onto the backup Server, inside the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System, I create a DWORD named LocalAccountTokenFilterPolicy with a Value of 1.

Then I try the same steps again. Nope. Permission still denied. At this point, I get suitably annoyed at VMware’s barely functional client-server software as well as the vagaries of UAC, and decide to just install vCenter Converter Standalone over at the Hyper-V box instead, just to make the netpain stop. If that works, I decide I can just roll back the firewall rule and disabling stealth mode.

Copy installer, paste installer. Wait. Double-click installer. Yes to UAC. Wait while mesmerized by 1990s style installer splash screen. Next, next, agree, next, next, next, next, install. Run standalone client now? Yes. Finish.

Then, I try it again. Convert machine, powered off, Hyper-V server, localhost, username, password, next. Permission denied. Argh. Oh, wait, the installer didn’t launch the converter elevated.

Elevate, repeat steps, still broken. Argh. Curse at the screen for a while, and then realise that I’m stupid. It’s not going to matter if I run the converter itself elevated, because it doesn’t try to use my network creds in a pass-through mode. So, as another stupid simple way, I try using the built-in Administrator account, which because of compatibility reasons (such as stupid applications like this one) does not have UAC enabled.

After that, I’m able to proceed. Seriously, am I really supposed to be crippling the entire security model of Windows just to convert a VM? Anyway, I roll back my registry hacks and firewall rules from above and retry, making sure they’re not neccessary as long as using the built-in Administrator.

After that, I just breeze through the wizard. I make sure the network card is connected to an isolated network, choose some appropriate storage in vSphere, and submit the job, and it seems to be running. Success seems within grasp! Estimated time remaining: 1 hours and 3 minutes. I decide to do some actual work, such as preparing for an after-hours P2V of a dying Windows Server 2003 SBS. (The irony is, we’re already in the process of moving him to a Windows Server 2012 R2 environment, but the server’s so flaky we can’t even get the file share data out of there reliably without his disks timing out… but that’s another article.)

34 minutes later, the VM conversion has finished. I look over in the vSphere Web Client and confirm that the VM is actually on an isolated network. I power on the VM, open the console, and grab figurative popcorn. “Getting devices ready”… hmm… seems there’s some kind of driver injection going on… and then, with no further ado, I get to the login screen. I check the vSphere Client to see if VMware tools is installed – nope. Not surprised not really disappointed, it would have been really cool if vCenter Converter brought them in, but that would just be asking too much.

I log onto the machine with the local admin account. First thing, I look at the network card. It’s there, but set to DHCP. Static IP settings are missing. Not surprising, since it’s a new network card. That’s all right, I just type in some new settings. I type in the same settings as the old VM had, expecting problems with hidden devices. And yes – that’s the case. “Already assigned to another adapter”. I click past it and proceed.

At this point, I’m happy enough to call the restore a success. It boots, and everything important (disks and network) seems to work, as well as the size of the VM (cores, RAM, disk layout etc) being appropriately set as by magic. Very convenient considering. I just kick off an install of VMware tools, making sure that wouldn’t cause any problems, and that’s when I reach some minor netpain:

Call “VirtualMachine.MountToolsInstaller” for object “server01.contoso.com” on vCenter Server “VCENTER” failed.
vix error code = 21002
This virtual machine does not have a CD-ROM drive configured. A virtual machine must have a CD-ROM drive configured for this option to work.

I just hot-add an optical drive into the VM, without rebooting. (Sometimes I love vSphere), and try again. VMware Tools installs like a champ, and I get to reboot to finalize the install. Nice!

Finally, I get to log in to the backup server, open up the Hyper-V manager, and delete the intermediate VM that was registered in Hyper-V. It has served its purpose. Also, can’t remember deleting the VHDX files, which keep the actual data, because deleting the VM does not also delete the virtual hard disks.

So… once this is all set up, this is my workflow for spinning up a Hyper-V VM backed up by Veeam on our vSphere setup:

1. Perform a Veeam restore of the Hyper-V VM onto the Hyper-V role on the backup server itself. Use suitable scratch space on the local server.
2. Run VMware vCenter Converter, to convert an offline VM from Hyper-V server. Source server localhost, make sure to use the built-in local administrator account (not any old admin account) due to UAC shenanigans.
3. Target your vCenter server, and restore onto suitable storage in the cluster.
4. Boot up the VM, add a CD-ROM drive, install VMware tools, reboot.
5. Set up proper IP addressing, reboot again.
6. Test, and bring into production.
7. Nuke the intermediate Hyper-V VM on the backup server. Make sure to also delete the associated VHDX files.

Entirely doable, although it involves two copies of the entire VM data, it’s not exactly efficient. But it’s good enough for now. 🙂

Licensing is a pain. Not so much for the sake of cost, but more for the sake of never quite knowing that you’re compliant.

I was considering a case today. What if one of my customers wants to grant administrative access to a vendor to one of their servers for the purposes of setting up and administering an on-premise application?

My first instinct was yes – they do. After all, they are a user that needs to log in. But then I thought a bit more about it. CALs are after all assigned to physical persons or devices, I can’t just go say “I hereby assign this CAL to Vendor X’s support team”, because that’s not a physical person. Would I have to ask for specific names of the vendor’s technicians for the purposes of license assignment? And how would this be enforced? Maybe just creating a few named users and then just disclaiming knowledge of whether they share their personal accounts, making it their problem? No. There had to be some kind of exception.

So I go out on the Internet, and I found this slightly helpful but mostly unhelpful post from the Microsoft Volume Licensing Blog. As witnessed by the comments, the blog entry raises more questions than it answers. It’s not really the fault of the blog, licensing is hard, really for no good reason other than to make work for licensing lawyers.

This is what they write:

7 – Do I need CALs for my administrators?

Server software licensed using CALs permits up to 2 users or devices to access the server software for the purposes of administration without CALs. However, if your administrators also use the software for anything other than administration (for example, they check their email), CALs will be required for them as well.

You could be forgiven to still fret and say “Oh, okay, so if my vendor has 10 employees I only need 8 CALs” and open your pocketbook and still have the administrative nightmare about having to know about who they employ and manage CAL assignment, adding the two “free” CALs to your pool of CALs.

But that’s actually not the case. If you read what Microsoft said, they specifically say “Server software licensed using CALs permits up to 2 users or devices to access the server software for the purposes of administration without CALs.” (emphasis mine).

Why is this important? Well, because the rules regarding CAL assignments do not come into play! The only way I could reasonably interpret this would be “permits up to 2 users or devices at the same time“. After all, there is no permanent assignment of CALs going on.

The relevant language from Microsoft’s Product Terms does not contradict this interpretation:

16. Administrative and Support Rights
Customer may allow access to server software running in any permitted OSE by two users without CALs solely for administrative purposes. Customer may also allow remote access to other Products solely for purposes of providing technical product support to Licensed Users or on Licensed Devices.

So, my conclusion is, I’m okay. We don’t need to assign CALs ever to users who only work as admins on the server, as long as no more than two of them are ever working on the same time on any given server.

Now, I’d like to point out for the record. I’m not a lawyer, I’m not a licensing expert. I don’t work for a licensing company.

This is not licensing advice. This is just my opinion about how I interpret the licensing rules. After all, I’m just a sysadmin annoyed with the various “pain in the nets” and blogging about it.

And I find that this obfuscated interpretation lines up with common sense. So that’s the interpretation I’m going with.