Running OpenVMS x86 on Proxmox — Common Pitfalls (and My Fix)

Welcome to another entry in CoffeeMuse’s Musings — where the coffee is strong, the curiosity is caffeinated, and the virtual machines don’t always cooperate.

I recently decided (with some encouragement from my cat’s disapproving stare) to try running OpenVMS x86-64 on Proxmox. Why? Because Proxmox is a fantastic, flexible hypervisor with a slick web interface, and OpenVMS… well, it’s OpenVMS , a beautiful relic of engineering that refuses to die. While Proxmox isn’t officially supported by VSI, it runs on KVM, which is supported. So in theory, it should work. Naturally, that means it didn’t. At least, not at first.

While troubleshooting (and sipping an unreasonable amount of espresso), I dove deep into documentation, forums, and community posts. I started to notice the same handful of issues cropping up for others, too. So I’ve rounded up the most common ones and compiled them here.

If you’re looking for step-by-step instructions with screenshots, and hopefully fewer cat interjections, check out my full guide: A Complete Guide to Running OpenVMS on Proxmox.

1. CPU Instruction-Set Mismatches

Symptom: Installation or boot halts with an error such as

No XSAVE instruction

Cause: OpenVMS x86-64 V9.2-3 requires several CPU features—including XSAVE—to be exposed to the guest. By default, Proxmox may select a generic CPU model (e.g., kvm64 or x86-64-v2) that doesn’t include XSAVE or related extensions unless the underlying hardware supports them.

Solution:

• In the VM’s Hardware → CPU settings, change the CPU type to host (passthrough), which maps all host CPU features directly into the guest.
• Verify your physical CPU meets VSI’s minimum requirements: SSE4.1, VT-x, XSAVE, NX bit, APIC, MTRR, and a 64-bit instruction set.

2. BIOS vs. UEFI Configuration

Symptom: VM fails to boot at all, or installer never appears, when using the default “BIOS” firmware.

Cause: OpenVMS x86-64 does not support legacy (SeaBIOS) boot. The installer and the system firmware rely on UEFI.

Solution:

• In the VM’s Hardware → BIOS settings, select OVMF (UEFI).
• Confirm the chipset is set to Q35 (not i440FX) to match VSI recommendations for KVM guests.

3. Insufficient Memory

Symptom: Installer aborts partway through or reports out-of-memory errors; bootloader fails to find a valid system disk.

Cause: VSI specifies minimum resource thresholds for a successful install.

Solution:

• Memory: Allocate at least 8 GB of RAM to the VM during installation.
• If memory is too low, the installer will refuse to proceed.

4. Disk Bus and Controller Misconfiguration

Symptom: VM boots into OpenVMS “Boot Manager” but fails to locate the system volume; or QEMU exits with a generic disk error.

Cause: Improperly attaching the OpenVMS VMDK to the VM (e.g., selecting unsupported controller interface) can prevent the guest from seeing its system volume.

Solution:

1. Import the provided OpenVMS VMDK into Proxmox storage:

qm disk import <VMID> X86_V923-community.vmdk <storage-name>

2. Attach it as a SCSI disk (scsi0 or scsi1) and select VirtIO-SCSI as the controller.
3. Set the VM’s boot order to boot from the correct SCSI disk (--boot order=scsi0).

5. Network Adapter Compatibility

Symptom: After boot, OpenVMS networking services (DECnet, TCP/IP) cannot find any interface or hang.

Cause: By default, Proxmox may use an unsupported virtual NIC model. OpenVMS x86-64 only supports specific network devices.

Solution:

• In Hardware → Network Device, choose virtio (the para-virtualized NIC) for KVM guests.
• Alternatively, select an emulated Intel NIC (e1000) which VSI also certifies.
• Do not manually set DECnet MAC addresses; let the hypervisor assign them automatically.

6. Console Communication

Symptom: No console output appears, or keyboard input seems unresponsive during installation.

Cause: OpenVMS x86-64 supports only a single serial port console by default; it won’t show output on the standard Proxmox VGA console.

Solution:

• In Hardware → Serial Port, add a Serial Port.
• As of V9.2-3, you can also enable the new “Guest Console” mode via the VSI Boot Manager:

BOOTMGR> OPA0

to interact through QEMU’s stdio console.

Conclusion

Running OpenVMS on Proxmox may not be officially supported, but with the right setup, a little determination, and the occasional espresso-fueled troubleshooting session, it’s absolutely doable. Hopefully this roundup helps save you from chasing the same issues I encountered.

Despite the challenges, I eventually managed to get OpenVMS x86 running reliably and repeatably on Proxmox.

The solution involved carefully configuring the virtual machine (correct UEFI boot, serial console access, disabling HPET, etc.), importing the VMDK correctly, and attaching everything in the right order.

For a complete walkthrough, don’t forget to check out my full guide: A Complete Guide to Running OpenVMS on Proxmox.

Until next time, may your logs be clean, your consoles responsive, and your cat merciful.

— CoffeeMuse

Additional Information

• VSI OpenVMS x86-64 v9.2-3 Installation Guide