Instead of running a machine dedicated entirely to ESXi, it is also possible to run it in a virtual machine. I found bgulla’s tutorial very helpful.

My R720 has Intel CPUs so I ran the following command to check to see if V-in-V support was enabled.

cat /sys/module/kvm_intel/parameters/nested

I got an N so it was disabled so I ran the following commands:

echo 'options kvm ignore_msrs=y' >> /etc/modprobe.d/kvm-intel.conf
echo 'options kvm-intel nested=Y ept=Y' >> /etc/modprobe.d/kvm-intel.conf
reboot

After the reboot I ran the following command again and got a Y instead.

cat /sys/module/kvm_intel/parameters/nested

I created a new VM with everything default except the following:

and then ran the following command in the Proxmox shell:

echo 'args: -machine vmport=off' >> /etc/pve/qemu-server/106.conf

Booting up was the standard ESXi installer. I was having issues with pressing F11 in the VNC session and found that pressing FN + COMMAND + F11 worked.