Description of problem:
Upstream bug: https://bugs.launchpad.net/nova/+bug/1469179
Nova seems to be reporting local disk for NFS backed cinder volumes. This is a problem trying to create VMs in the overcloud. Found found references to this bug for juno, NOT kilo, then it seems to be back for liberty.
https://ask.openstack.org/en/question/65854/incorrect-advertisement-of-block-storage-total-and-free-space/?answer=92450#post-id-92450https://ask.openstack.org/en/question/88772/if-i-boot-from-a-volume-cinder-why-appears-as-im-using-computes-local-disk-usage/https://bugs.launchpad.net/nova/+bug/1469179
~~~
[stack@controller1 ~]$ nova hypervisor-stats
+----------------------+---------+
| Property | Value |
+----------------------+---------+
| count | 2 |
| current_workload | 0 |
| disk_available_least | 396 |
| free_disk_gb | -300 |
| free_ram_mb | 950218 |
| local_gb | 400 |
| local_gb_used | 700 |
| memory_mb | 1031114 |
| memory_mb_used | 80896 |
| running_vms | 14 |
| vcpus | 64 |
| vcpus_used | 25 |
+----------------------+---------+
~~~
~~~
[heat-admin@compute1 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 279G 26G 254G 10% /
devtmpfs 252G 0 252G 0% /dev
tmpfs 252G 54M 252G 1% /dev/shm
tmpfs 252G 1.4M 252G 1% /run
tmpfs 252G 0 252G 0% /sys/fs/cgroup
tmpfs 51G 0 51G 0% /run/user/991
10.29.104.17:/osp7cf_glance 500G 23G 478G 5% /var/lib/glance/images
tmpfs 51G 0 51G 0% /run/user/990
10.29.104.27:/osp7cf_cinder3 820G 1.5G 818G 1% /var/lib/cinder/mnt/7476a0c43f2f41a0b2b5b39afaf2f095
10.29.104.15:/osp7cf_cinder2 820G 8.4G 811G 2% /var/lib/cinder/mnt/fbdef210720e36d661073a6e8ee114ab
10.29.104.15:/osp7cf_cinder1 820G 21G 799G 3% /var/lib/cinder/mnt/5ea493792ed71f306f40d94699619c1a
10.29.104.27:/osp7cf_cinder4 820G 1.4G 818G 1% /var/lib/cinder/mnt/57128760568a05d1c0e3e78b6ed64817
tmpfs 51G 0 51G 0% /run/user/1000
~~~
Additional info:
From the upstream bug report:
~~~
instance.root_gb means size of host local disk used by the instance root disk. All code that uses this attribute treats it that way.
Volume-backed instances have root disks placed in Cinder. Therefore obviously root_gb should be 0 for them. However currently this is not so. The same applies to min_disk and size image attributes, used for boot from a volume.
As a result the code which uses these attributes works incorrectly. Some problems are already detected [1], other are not yet [2].
There are two kinds of bugs:
1 Nova fails to launch an instance from a large volume if the volume size (or an original image's min_disk) is greater than requested flavor.root_gb.
2 Nova incorrectly calculates host disk space consumed by volume-backed instances.
To fix all these problems fully it is proposed to set root_gb, min_disk, size to 0 for volume-backed instances.
[1] https://bugs.launchpad.net/nova/+bug/1334974https://bugs.launchpad.net/nova/+bug/1459491https://bugs.launchpad.net/nova/+bug/1466305https://bugs.launchpad.net/nova/+bug/1457517https://bugs.launchpad.net/nova/+bug/1358566
[2] https://github.com/openstack/nova/blob/master/nova/notifications.py#L407https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L250
~~~