How to Install VMware ESXi on a VirtualBox VM

Installing VMware ESXi inside a VirtualBox virtual machine is a common setup for learning, testing, and lab environments. It allows you to experiment with ESXi without dedicating a physical server. However, because ESXi itself is a bare-metal hypervisor, running it inside another hypervisor requires some extra configuration.

The good news is that with the right settings enabled, ESXi works reliably inside VirtualBox for practice and non-production use.

Install VMware ESXi on a VirtualBox VM

The steps below walk you through the complete process, from preparing VirtualBox to successfully booting and installing ESXi. We recommend following them in order, as ESXi is very strict about virtualization requirements.

1. Enable Hardware Virtualization on Your PC

Before touching VirtualBox, you need to make sure your system supports nested virtualization.

1. Restart your PC and enter BIOS/UEFI settings.
2. Locate CPU or Advanced settings.
3. Enable Intel VT-x or AMD-V (SVM), depending on your processor.
4. Save changes and boot back into Windows.
5. In Windows, open Task Manager and confirm virtualization is enabled.

Without hardware virtualization, ESXi will not boot inside VirtualBox.

2. Install and Update VirtualBox

You must use a recent version of VirtualBox to ensure proper nested virtualization support.

1. Download and install VirtualBox from the official website.
2. Install the matching Extension Pack for the same version.
3. Restart your PC after installation.

Keeping VirtualBox updated reduces boot and compatibility issues with ESXi.

3. Create a New Virtual Machine for ESXi

Now you’ll create the VM that will host ESXi.

1. Open VirtualBox and click New.
2. Name the VM (for example, ESXi Lab).
3. Set Type to Other.
4. Set Version to Other/Unknown (64-bit).
5. Allocate at least 4 GB RAM (8 GB recommended).
6. Assign 2 or more CPU cores.

ESXi needs adequate CPU and memory to complete installation and boot properly.

4. Configure Processor and Nested Virtualization

This is the most important step for running ESXi inside VirtualBox.

1. Select the VM and click Settings.
2. Go to System > Processor.
3. Increase CPU cores to at least 2.
4. Check Enable PAE/NX.
5. Enable Nested VT-x/AMD-V.
6. Go to System > Motherboard and enable EFI.

Without nested virtualization, ESXi will fail with CPU or VT-x errors.

5. Configure Storage and Network

Proper storage and networking settings help ESXi detect hardware correctly.

1. In Settings > Storage, attach the ESXi ISO as an optical disk.
2. Use a VDI or VMDK disk with at least 40 GB size.
3. In Network, select Bridged Adapter (recommended for lab access).
4. Choose your active network interface.

Bridged networking allows you to access ESXi via browser later.

6. Apply Required VBoxManage Commands

ESXi requires specific CPU identifiers that VirtualBox doesn’t expose by default.

1. Close VirtualBox completely.
2. Open Command Prompt as Administrator.
3. Run the following commands (replace VM name if needed): VBoxManage modifyvm "ESXi Lab" --cpu-profile "Intel Core i7-6700K"
4. Restart VirtualBox.

This step prevents ESXi from failing CPU compatibility checks.

7. Start the VM and Install ESXi

You’re now ready to install ESXi inside the VM.

1. Start the VirtualBox VM.
2. When the ESXi installer loads, press Enter to begin.
3. Accept the license agreement.
4. Select the virtual disk as the installation target.
5. Choose keyboard layout and set a root password.
6. Complete installation and reboot the VM.
7. Remove the ESXi ISO when prompted.

After reboot, ESXi should load successfully.

8. Access ESXi Web Interface

Once ESXi boots, you can manage it from a browser.

1. Note the IP address shown on the ESXi console.
2. Open a browser on your host PC.
3. Enter the IP address in the address bar.
4. Log in using root and the password you created.

You now have a fully functional VMware ESXi lab running inside VirtualBox.

Wrapping Up

Installing ESXi on a VirtualBox VM is an excellent way to learn virtualization, test configurations, and build a home lab without dedicated hardware. While the setup requires extra steps like enabling nested virtualization and applying CPU profiles, the end result is well worth it.

Keep in mind that this setup is intended for testing and learning only, not production use. Once everything is configured correctly, ESXi runs surprisingly well inside VirtualBox and gives you hands-on experience with enterprise-grade virtualization from a single PC.