When you buy an HPE ProLiant server, it comes with HPE iLO (Intelligent Lights-out) by default. iLO is a remote management solution, allowing you to remotely control your server, allowing for many actions, including turning the server on and off, remotely accessing the server’s keyboard, mouse and display (as if you were plugged in directly to the server), or mounting (and booting off) ISO files. In the industry, such functionality is generally also known as IPMI, named after the IPMI standard which provides a vendor-neutral standard for this type of system management.
This is a very convenient feature of course for remote management. But it’s also important that the feature is properly secured. Else, the feature can be abused. For example, an attacker could turn your server off, or even restart it with a malicious ISO file mounted, and install malware on your machine. One example of such attacks were the 2018 attacks by a group called JungleSec, which installed ransomware on poorly secured servers through similar features on competing vendors, some of which allow logins using well-known default credentials.
Fortunately, HPE iLO is a cut above many of the competitors offering IPMI solutions, in that HPE will randomize a unique password for every server for the iLO. So, even if you never touch the iLO funcitionality, you should be reasonably well protected, right?
That’s a reasonable assumption to make, unfortunately, it doesn’t turn out to be true in all cases.
IPMI isn’t just a generic name for iLO-like functionality, it’s also a vendor-neutral standard. Unfortunately, baked right into the standard is a pretty glaring design flaw, which is present in all compliant implementations.
I’m not going to get into the weeds with juicy details. All you need to know for the purpose of this post, is that there’s a flaw in the IPMI 2.0 specification, that allows an attacker to obtain password hashes for any IPMI account, as long as they know the username for it. The vulnerability is baked into the IPMI specification, and cannot be patched.
So, therefore, if you can talk to the IPMI service of an HPE ProLiant server, and you know it’s username, you can dump out a hash of the password for the iLO service.
Unfortunately, the username on all HPE ProLiant servers is the same, and is well known, so getting a copy of the password hash is pretty easy, allowing for easy password cracking.
Even more unfortunately, the password generation algorithm used by HPE to set their iLO passwords, do not produce passwords strong enough to resist cracking. In my own tests experiments, I was able to reliably crack a password in less than 30 minutes using a contemporary gaming desktop computer (Geforce RTX 3070, 1600 MH/s), with no more information than what pattern HPE uses for their passwords. A sophisticated attacker might use a more powerful machine, major cloud vendors provide machines with 10x the cracking horsepower for a very practically affordable $25 per hour, making cracking a single HP iLO password cost less than $1 in computing power.
For your server to be vulnerable to this particular attack, a few things have to line up:
- iLO must be reachable on your network. On some servers, iLO will piggyback on your main network card, and use DHCP. You might have iLO on your network and not even know it. On other servers, iLO only runs on a seperate dedicated network card.
- You must be able to reach it on UDP port 623, which is the port used by IPMI. As far as I know, it’s not possible to exploit this if only the web interface is exposed. Your firewall may protect you from attacks from remote networks, but unfortunately, a common practice is to put the server directly on your office LAN, allowing an attacker to pivot from a compromized PC to attack your server, which would bypass your firewall. For protection on the LAN side, you may be able to leverage Acess Control Lists on your switches.
- The IPMI service must be enabled. This setting (called IPMI/DCMI over LAN) is fortunately disabled by default on modern servers (Gen10 and newer), but is ENABLED on older hardware (Gen9 and older).
- The username must be guessable. On HPE servers, the default username is common across all servers.
- The password must be crackable. The default passwords used by HPE are crackable using affordable hardware in a short amount of time.
- Access to iLO must allow you to compromize the host system in a meaningful way. This is almost always true unless you have a robust security configuration.
In order to mitigate this, you need to do at least ONE of the following things. You don’t need to do everything, of course, and you probably don’t want to do everything on this list. What mitigations you end up putting in place will depend on your specific requirements.
- If you don’t need iLO, consider completely disabling the functionality. Personally, I wouldn’t do this, because iLO is legitimately far too useful to disable. You should still use firewalls and network segmentation to make sure only specific, trusted computers can reach the iLO interface.
- If you don’t want to lock down access to the iLO web interface too hard, you can still protect yourself by locking down access to UDP port 623. That will stop this attack in its tracks, even if the web interface is accessible. Remember, your firewall will not protect your ILO card from attackers on the same LAN.
- If you don’t need access to industry-standard (and unsecure) IPMI, consider disabling IPMI/DCMI over LAN entirely. That will also stop this attack in its tracks, and you can still manage iLO using the web interface and through HPE’s own tools.
- If you must have a weak password, consider changing the username, and guard the username as a close secret. The username is required to do a dump of the password hashes through IPMI.
- Use a long, secure password for iLO. I would suggest at least a 15-character password containing random uppercase, lowercase, and digits.
- If you really must run IPMI, accessable from untrusted devices, and you really must have a weak guessable username, and a crackable password, your last line of defense would be implementing BIOS passwords to prevent an attacker from booting off removable media. You will also need to be aggressive about locking your screen and/or logging off from the console when the server is not in use. Even in these cases, it’ll be possible to remotely shut down your server. It’s not clear how effective this would be in the long run to stop a determined attacker, but it’s worth considering as a last line of defense.
Finally, I’d like to point out that the ability to dump passwords out of IPMI boxes is not news. Dan Farmer wrote about this in 2013. But even though it’s not news, it’s still a current vulnerability, and because the vulnerability is in the standard, we just have to learn to live with it.
What I feel I bring to the table with this article is to raise awareness of this bug, as well as HPE’s default passwords are not strong enough to resist this type of attack, and in fact are crackable in minutes. I’ve always been impressed that HPE goes above their competition and set strong passwords by default, unlike other vendors, but unfortunately, they don’t stand up to this kind of attack. To HPE’s credit, modern servers have IPMI over LAN disabled by default, and modern servers will also strongly encourage you to change default passwords.
Because my mission is to help server administrators defend their systems, not to write a HOWTO for would-be attackers, I will not be provide details as to default usernames used by iLO, or the specific pattern used by HPE to generate passwords. This information is not useful to defenders, and would only serve to help attackers. I will also not name the specific tools used to pull off this attack, but they are publically and readily available with a little bit of research. It’s not difficult to get this information elsewhere, but I will not be the one providing it to you, nor will I approve any comments requesting or providing such information.
Stay safe out there!
References:
A Penetration Tester’s Guide to IPMI and BMCs (HD Moore, Rapid 7, Jul 2 2013)
Cracking IPMI Passwords Remotely (Dan Farmer, fish2.com, date unknown, circa 2013)