ESXi Update fails with "[Errno 28] No space left on device"

This appears to be a somewhat recent problem, and when searching for an answer, a lot of posts reference solutions or VMWare KB's that don't help (enabling swap, checking inodes, etc.).

This problem appeared somewhere between ESXi-6.7.0-20190104001-standard (Build 11675023) and ESXi-6.7.0-20190404001-standard (Build 13473784). When updating via CLI, you get the following error message:

[root@localhost:~] esxcli software profile update -p ESXi-6.7.0-20190504001-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
 [InstallationError]
 [Errno 28] No space left on device
       vibs = VMware_locker_tools-light_10.3.5.10430147-12986307
 Please refer to the log file for more details.

TinkerTry is the only place where I've seen the (for me) correct solution posted: https://tinkertry.com/easy-update-to-latest-esxi#apr-14-2019-update. Basically, update VMware_locker_tools-light_10.3.5.10430147-12986307.vib (see commands below or in TinkerTry) and everything seems to work fine. I also enabled swap on my datastore, but this didn't seem to help. However, if locker tools vib doesn't help, it may be a combination of both.

EDIT/UPDATE: I ran into this again upgrading to ESXi-6.7.0-20200403001-standard - The fix was the same, but I had to upgrade to an even newer version of VMware_locker_tools-light. To find the latest version, use the following:

esxcli software sources vib list --depot=https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml | grep tools-light | sort

You can add a "grep -v Downgrade" to get only relevant versions. Replace the version number in the wget, and install like normal. The current latest version is VMware_locker_tools-light_11.0.5.15389592-15843807.vib

</Update>

EDIT/UPDATE #2:

I received another no space left on device message when updating to 7.0.1-0.0.16850804. While troubleshooting what appeared to be a failure to install though, I noticed a "Reboot is pending from previous transaction.". After rebooting, the web interface said I was on the latest/correct version, so appearently everything worked?

 [InstallationError]
 Failed to update bootloader: [Errno 28] No space left on device
      cause = [Errno 28] No space left on device
       vibs = ['VMware_bootbank_esx-base_7.0.1-0.0.16850804']
 Please refer to the log file for more details

Re-running the update command after a reboot showed "Host is not changed".

</Update>

[root@localhost:~] cd /tmp
[root@localhost:/tmp] wget http://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/tools-light/VMware_locker_tools-light_10.3.5.10430147-12986307.vib
Connecting to hostupdate.vmware.com (23.54.148.48:80)
VMware_locker_tools- 100% |**********************************************************************************************************************************|   153M  0:00:00 ETA
[root@localhost:/tmp] esxcli software vib install -f -v /tmp/VMware_locker_tools-light_10.3.5.10430147-12986307.vib
Installation Result
   Message: Operation finished successfully.
   Reboot Required: false
   VIBs Installed: VMware_locker_tools-light_10.3.5.10430147-12986307
   VIBs Removed: VMware_locker_tools-light_10.3.2.9925305-10176879
   VIBs Skipped:
[root@localhost:/tmp]

<reboot><reenable ssh>

[root@localhost:~] esxcli software profile update -p ESXi-6.7.0-20190504001-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
Update Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
[...snip...]
[root@localhost:~]

For my own later reference, here's the quick steps guide to upgrading ESXi. If you aren't familiar with this process, please read more (https://tinkertry.com is a good source) before attempting an update and fully understand the process.

1. Suspend all guests
2. Enter Maintenance Mode
3. Reboot
4. Start SSH Service
5. List available updates: esxcli software sources profile list --depot=https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml | sort The latest update should be near the bottom, but may not be the very bottom! *
6. Apply the latest version: esxcli software profile update -p ESXi-6.7.0-20190504001-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml <- Replace the version after -p with the version you found via the above list
7. Reboot again
8. Exit Maintenance Mode
9. Restart guests

* If, like me, your DNS server lives on your ESXi box, you can `echo "nameserver 1.1.1.1" > /etc/resolv.conf` to temporarily use an external DNS during the upgrade process.