Bug 1464182

Summary: openstack-nova: unable to launch an instance: InternalError: Unable to get host UUID: /etc/machine-id is empty
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-novaAssignee: Martin André <m.andre>
Status: CLOSED WORKSFORME QA Contact: Alexander Chuzhoy <sasha>
Severity: high Docs Contact:
Priority: urgent    
Version: 12.0 (Pike)CC: berrange, dasmith, eglynn, kchamart, m.andre, ohochman, sbauza, sferdjao, sgordon, srevivo, tvignaud, vromanso
Target Milestone: rcKeywords: Triaged
Target Release: 12.0 (Pike)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-31 16:16:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
/var/log/containers/nova/nova-compute.log none

Description Alexander Chuzhoy 2017-06-22 15:10:12 UTC
openstack-nova: unable to launch an instance: InternalError: Unable to get host UUID: /etc/machine-id is empty

Environment:
openstack-nova-compute-16.0.0-0.20170616124201.8238e8a.el7ost.noarch
openstack-nova-scheduler-16.0.0-0.20170616124201.8238e8a.el7ost.noarch
python-novaclient-9.0.0-0.20170609003607.7585703.el7ost.noarch
openstack-nova-placement-api-16.0.0-0.20170616124201.8238e8a.el7ost.noarch
python-nova-16.0.0-0.20170616124201.8238e8a.el7ost.noarch
openstack-nova-migration-16.0.0-0.20170616124201.8238e8a.el7ost.noarch
openstack-nova-conductor-16.0.0-0.20170616124201.8238e8a.el7ost.noarch
openstack-nova-console-16.0.0-0.20170616124201.8238e8a.el7ost.noarch
openstack-nova-api-16.0.0-0.20170616124201.8238e8a.el7ost.noarch
puppet-nova-11.2.0-0.20170614050043.eeb0af1.el7ost.noarch
openstack-nova-common-16.0.0-0.20170616124201.8238e8a.el7ost.noarch
openstack-nova-novncproxy-16.0.0-0.20170616124201.8238e8a.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch
instack-undercloud-7.1.1-0.20170616135935.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170616123155.el7ost.noarch


Steps to reproduce:

1. Deploy overcloud with containers.
openstack overcloud deploy \
--templates /usr/share/openstack-tripleo-heat-templates \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /home/stack/virt/network/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/hostnames.yml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /home/stack/virt/docker-osp12.yaml \
--log-file overcloud_deployment_34.log


2. Try to launch an instance with:
openstack server create --flavor 1 --image cirros vm



Result:
(overcloud) [stack@undercloud-0 ~]$ nova list
+--------------------------------------+------+--------+------------+-------------+----------+
| ID                                   | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+----------+
| 87825739-a581-4884-b4d6-7aae3ae5976c | vm   | ERROR  | -          | NOSTATE     |          |
+--------------------------------------+------+--------+------------+-------------+----------+



Grepping the /var/log/containers/nova/nova-compute.log file on compute for errors:
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [req-da762255-769b-4b76-b58b-9ba4e5f6d058 c456491527bb4b25920a6b10819bf389 d17f34b03a794c4c9ffeac66382ea8f5 - default default] [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] Instance failed to spawn: InternalError: Unable to get host UUID: /etc/machine-id is empty
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] Traceback (most recent call last):
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2154, in _build_resources
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]     yield resources
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1960, in _build_and_run_instance
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]     block_device_info=block_device_info)
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2761, in spawn
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]     block_device_info=block_device_info)
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4973, in _get_guest_xml
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]     context)
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4792, in _get_guest_config
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]     root_device_name)
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4470, in _configure_guest_by_virt_type
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]     guest.sysinfo = self._get_guest_config_sysinfo(instance)
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3794, in _get_guest_config_sysinfo
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]     sysinfo.system_serial = self._sysinfo_serial_func()
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3783, in _get_host_sysinfo_serial_auto
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]     return self._get_host_sysinfo_serial_os()
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3777, in _get_host_sysinfo_serial_os
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]     raise exception.InternalError(msg)
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] InternalError: Unable to get host UUID: /etc/machine-id is empty
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]

Comment 1 Alexander Chuzhoy 2017-06-22 15:16:11 UTC
Created attachment 1290755 [details]
/var/log/containers/nova/nova-compute.log

Comment 2 Alexander Chuzhoy 2017-06-22 15:17:59 UTC
On compute node:
[root@compute-0 ~]# cat /etc/machine-id 
270d5597e0414f018ba9787924d7626b
[root@compute-0 ~]# docker exec -it nova_libvirt bash
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified



On container:
()[root@compute-0 /]# cat /etc/machine-id 
()[root@compute-0 /]#

Comment 3 Alexander Chuzhoy 2017-06-22 15:37:00 UTC
Was able to w/a this issue by placing a string in /etc/machine-id on the nova_compute container. It was the same string taken from the compute host.

Comment 4 Alexander Chuzhoy 2017-06-22 15:48:34 UTC
openstack-base-docker:2017-06-21.5 also has an empty /etc/machine-id file.

Comment 5 Kashyap Chamarthy 2017-06-23 13:10:32 UTC
We need to first figure out what *exactly* Nova is using the `/etc/machine-id` for.

So far, we can see that Docker is creating the /etc/machine-id, looking at this bug (thanks to Ollie Walsh for the pointer):

    https://bugzilla.redhat.com/show_bug.cgi?id=1130498

Ollie Walsh says this should probably be fixed in base RHEL / CentOS images.  As the base CentOS / RHEL images have a 0 byte 'machine-id'.

---

The official systemd documentation says, the `systemd-machine-id-setup` needs to be used to initialize the 'machine-id':

https://www.freedesktop.org/software/systemd/man/systemd-machine-id-setup.html


Also, from documentation of `machine-id(5)`:

    "This ID uniquely identifies the host. It should be considered
    "confidential", and must not be exposed in untrusted environments, 
    in particular on the network. If a stable unique identifier that is
    tied to the machine is needed for some application, the machine ID
    or any part of it must not be used directly. Instead the machine ID
    should be hashed with a cryptographic, keyed hash function, using a
    fixed, application-specific key. That way the ID will be properly
    unique, and derived in a constant way from the machine ID but there
    will be no way to retrieve the original machine ID from the
    application-specific one"

Comment 6 Kashyap Chamarthy 2017-06-23 14:33:04 UTC
(In reply to Kashyap Chamarthy from comment #5)
> We need to first figure out what *exactly* Nova is using the
> `/etc/machine-id` for.

So, looking at the code (nova/virt/libvirt/driver.py), these two methods _get_host_sysinfo_serial_os(), and _get_host_sysinfo_serial_auto() use /etc/machine-id.  Nova uses it (/etc/machine-id) for the 'sysinfo_serial' config attribute to the populate the host "serial" 'UUID exposed to guest in the virtual BIOS. Permitted options are "hardware", "os", "none" or "auto" (default):

[...]

    cfg.StrOpt('sysinfo_serial',
               default='auto',
               choices=('none', 'os', 'hardware', 'auto'),
               help='The data source used to the populate the host "serial" '
                    'UUID exposed to guest in the virtual BIOS.'),

[...]

Comment 7 Omri Hochman 2017-06-23 14:52:28 UTC
do we need to mount this file from the compute-node to the container?

Comment 8 Omri Hochman 2017-06-23 14:53:44 UTC
taking it back to container DFG, as the solution might be to remove this file from the container during prep , and it would recreate with the right content.

Comment 9 Alexander Chuzhoy 2017-08-31 16:16:56 UTC
The issue doesn't reproduce and I'm able to launch instances although the machine-id file is empty on container:

[root@overcloud-compute-0 ~]# cat /etc/machine-id 
f18324a2198a4534bb27b9d3af207b16


[root@overcloud-compute-0 ~]# docker exec -u root -it nova_libvirt bash
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
()[root@overcloud-compute-0 /]# cat /etc/machine-id 
()[root@overcloud-compute-0 /]# 


Closing the bug.