openstack-nova: unable to launch an instance: InternalError: Unable to get host UUID: /etc/machine-id is empty Environment: openstack-nova-compute-16.0.0-0.20170616124201.8238e8a.el7ost.noarch openstack-nova-scheduler-16.0.0-0.20170616124201.8238e8a.el7ost.noarch python-novaclient-9.0.0-0.20170609003607.7585703.el7ost.noarch openstack-nova-placement-api-16.0.0-0.20170616124201.8238e8a.el7ost.noarch python-nova-16.0.0-0.20170616124201.8238e8a.el7ost.noarch openstack-nova-migration-16.0.0-0.20170616124201.8238e8a.el7ost.noarch openstack-nova-conductor-16.0.0-0.20170616124201.8238e8a.el7ost.noarch openstack-nova-console-16.0.0-0.20170616124201.8238e8a.el7ost.noarch openstack-nova-api-16.0.0-0.20170616124201.8238e8a.el7ost.noarch puppet-nova-11.2.0-0.20170614050043.eeb0af1.el7ost.noarch openstack-nova-common-16.0.0-0.20170616124201.8238e8a.el7ost.noarch openstack-nova-novncproxy-16.0.0-0.20170616124201.8238e8a.el7ost.noarch openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch instack-undercloud-7.1.1-0.20170616135935.el7ost.noarch openstack-tripleo-heat-templates-7.0.0-0.20170616123155.el7ost.noarch Steps to reproduce: 1. Deploy overcloud with containers. openstack overcloud deploy \ --templates /usr/share/openstack-tripleo-heat-templates \ --libvirt-type kvm \ --ntp-server clock.redhat.com \ -e /home/stack/virt/network/network-environment.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/hostnames.yml \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \ -e /home/stack/virt/debug.yaml \ -e /home/stack/virt/nodes_data.yaml \ -e /home/stack/virt/docker-osp12.yaml \ --log-file overcloud_deployment_34.log 2. Try to launch an instance with: openstack server create --flavor 1 --image cirros vm Result: (overcloud) [stack@undercloud-0 ~]$ nova list +--------------------------------------+------+--------+------------+-------------+----------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------+--------+------------+-------------+----------+ | 87825739-a581-4884-b4d6-7aae3ae5976c | vm | ERROR | - | NOSTATE | | +--------------------------------------+------+--------+------------+-------------+----------+ Grepping the /var/log/containers/nova/nova-compute.log file on compute for errors: 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [req-da762255-769b-4b76-b58b-9ba4e5f6d058 c456491527bb4b25920a6b10819bf389 d17f34b03a794c4c9ffeac66382ea8f5 - default default] [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] Instance failed to spawn: InternalError: Unable to get host UUID: /etc/machine-id is empty 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] Traceback (most recent call last): 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2154, in _build_resources 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] yield resources 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1960, in _build_and_run_instance 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] block_device_info=block_device_info) 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2761, in spawn 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] block_device_info=block_device_info) 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4973, in _get_guest_xml 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] context) 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4792, in _get_guest_config 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] root_device_name) 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4470, in _configure_guest_by_virt_type 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] guest.sysinfo = self._get_guest_config_sysinfo(instance) 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3794, in _get_guest_config_sysinfo 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] sysinfo.system_serial = self._sysinfo_serial_func() 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3783, in _get_host_sysinfo_serial_auto 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] return self._get_host_sysinfo_serial_os() 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3777, in _get_host_sysinfo_serial_os 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] raise exception.InternalError(msg) 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] InternalError: Unable to get host UUID: /etc/machine-id is empty 2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]
Created attachment 1290755 [details] /var/log/containers/nova/nova-compute.log
On compute node: [root@compute-0 ~]# cat /etc/machine-id 270d5597e0414f018ba9787924d7626b [root@compute-0 ~]# docker exec -it nova_libvirt bash tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified On container: ()[root@compute-0 /]# cat /etc/machine-id ()[root@compute-0 /]#
Was able to w/a this issue by placing a string in /etc/machine-id on the nova_compute container. It was the same string taken from the compute host.
openstack-base-docker:2017-06-21.5 also has an empty /etc/machine-id file.
We need to first figure out what *exactly* Nova is using the `/etc/machine-id` for. So far, we can see that Docker is creating the /etc/machine-id, looking at this bug (thanks to Ollie Walsh for the pointer): https://bugzilla.redhat.com/show_bug.cgi?id=1130498 Ollie Walsh says this should probably be fixed in base RHEL / CentOS images. As the base CentOS / RHEL images have a 0 byte 'machine-id'. --- The official systemd documentation says, the `systemd-machine-id-setup` needs to be used to initialize the 'machine-id': https://www.freedesktop.org/software/systemd/man/systemd-machine-id-setup.html Also, from documentation of `machine-id(5)`: "This ID uniquely identifies the host. It should be considered "confidential", and must not be exposed in untrusted environments, in particular on the network. If a stable unique identifier that is tied to the machine is needed for some application, the machine ID or any part of it must not be used directly. Instead the machine ID should be hashed with a cryptographic, keyed hash function, using a fixed, application-specific key. That way the ID will be properly unique, and derived in a constant way from the machine ID but there will be no way to retrieve the original machine ID from the application-specific one"
(In reply to Kashyap Chamarthy from comment #5) > We need to first figure out what *exactly* Nova is using the > `/etc/machine-id` for. So, looking at the code (nova/virt/libvirt/driver.py), these two methods _get_host_sysinfo_serial_os(), and _get_host_sysinfo_serial_auto() use /etc/machine-id. Nova uses it (/etc/machine-id) for the 'sysinfo_serial' config attribute to the populate the host "serial" 'UUID exposed to guest in the virtual BIOS. Permitted options are "hardware", "os", "none" or "auto" (default): [...] cfg.StrOpt('sysinfo_serial', default='auto', choices=('none', 'os', 'hardware', 'auto'), help='The data source used to the populate the host "serial" ' 'UUID exposed to guest in the virtual BIOS.'), [...]
do we need to mount this file from the compute-node to the container?
taking it back to container DFG, as the solution might be to remove this file from the container during prep , and it would recreate with the right content.
The issue doesn't reproduce and I'm able to launch instances although the machine-id file is empty on container: [root@overcloud-compute-0 ~]# cat /etc/machine-id f18324a2198a4534bb27b9d3af207b16 [root@overcloud-compute-0 ~]# docker exec -u root -it nova_libvirt bash tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified tput: No value for $TERM and no -T specified ()[root@overcloud-compute-0 /]# cat /etc/machine-id ()[root@overcloud-compute-0 /]# Closing the bug.