Bug 1464182 - openstack-nova: unable to launch an instance: InternalError: Unable to get host UUID: /etc/machine-id is empty
openstack-nova: unable to launch an instance: InternalError: Unable to get ho...
Status: CLOSED WORKSFORME
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
urgent Severity high
: rc
: 12.0 (Pike)
Assigned To: Martin André
Alexander Chuzhoy
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-22 11:10 EDT by Alexander Chuzhoy
Modified: 2017-08-31 12:16 EDT (History)
13 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-31 12:16:56 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
/var/log/containers/nova/nova-compute.log (29.66 KB, application/x-gzip)
2017-06-22 11:16 EDT, Alexander Chuzhoy
no flags Details

  None (edit)
Description Alexander Chuzhoy 2017-06-22 11:10:12 EDT
openstack-nova: unable to launch an instance: InternalError: Unable to get host UUID: /etc/machine-id is empty

Environment:
openstack-nova-compute-16.0.0-0.20170616124201.8238e8a.el7ost.noarch
openstack-nova-scheduler-16.0.0-0.20170616124201.8238e8a.el7ost.noarch
python-novaclient-9.0.0-0.20170609003607.7585703.el7ost.noarch
openstack-nova-placement-api-16.0.0-0.20170616124201.8238e8a.el7ost.noarch
python-nova-16.0.0-0.20170616124201.8238e8a.el7ost.noarch
openstack-nova-migration-16.0.0-0.20170616124201.8238e8a.el7ost.noarch
openstack-nova-conductor-16.0.0-0.20170616124201.8238e8a.el7ost.noarch
openstack-nova-console-16.0.0-0.20170616124201.8238e8a.el7ost.noarch
openstack-nova-api-16.0.0-0.20170616124201.8238e8a.el7ost.noarch
puppet-nova-11.2.0-0.20170614050043.eeb0af1.el7ost.noarch
openstack-nova-common-16.0.0-0.20170616124201.8238e8a.el7ost.noarch
openstack-nova-novncproxy-16.0.0-0.20170616124201.8238e8a.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch
instack-undercloud-7.1.1-0.20170616135935.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170616123155.el7ost.noarch


Steps to reproduce:

1. Deploy overcloud with containers.
openstack overcloud deploy \
--templates /usr/share/openstack-tripleo-heat-templates \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /home/stack/virt/network/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/hostnames.yml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /home/stack/virt/docker-osp12.yaml \
--log-file overcloud_deployment_34.log


2. Try to launch an instance with:
openstack server create --flavor 1 --image cirros vm



Result:
(overcloud) [stack@undercloud-0 ~]$ nova list
+--------------------------------------+------+--------+------------+-------------+----------+
| ID                                   | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+----------+
| 87825739-a581-4884-b4d6-7aae3ae5976c | vm   | ERROR  | -          | NOSTATE     |          |
+--------------------------------------+------+--------+------------+-------------+----------+



Grepping the /var/log/containers/nova/nova-compute.log file on compute for errors:
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [req-da762255-769b-4b76-b58b-9ba4e5f6d058 c456491527bb4b25920a6b10819bf389 d17f34b03a794c4c9ffeac66382ea8f5 - default default] [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] Instance failed to spawn: InternalError: Unable to get host UUID: /etc/machine-id is empty
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] Traceback (most recent call last):
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2154, in _build_resources
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]     yield resources
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1960, in _build_and_run_instance
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]     block_device_info=block_device_info)
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2761, in spawn
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]     block_device_info=block_device_info)
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4973, in _get_guest_xml
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]     context)
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4792, in _get_guest_config
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]     root_device_name)
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4470, in _configure_guest_by_virt_type
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]     guest.sysinfo = self._get_guest_config_sysinfo(instance)
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3794, in _get_guest_config_sysinfo
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]     sysinfo.system_serial = self._sysinfo_serial_func()
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3783, in _get_host_sysinfo_serial_auto
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]     return self._get_host_sysinfo_serial_os()
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3777, in _get_host_sysinfo_serial_os
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]     raise exception.InternalError(msg)
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c] InternalError: Unable to get host UUID: /etc/machine-id is empty
2017-06-22 10:56:36.617 1 ERROR nova.compute.manager [instance: 87825739-a581-4884-b4d6-7aae3ae5976c]
Comment 1 Alexander Chuzhoy 2017-06-22 11:16 EDT
Created attachment 1290755 [details]
/var/log/containers/nova/nova-compute.log
Comment 2 Alexander Chuzhoy 2017-06-22 11:17:59 EDT
On compute node:
[root@compute-0 ~]# cat /etc/machine-id 
270d5597e0414f018ba9787924d7626b
[root@compute-0 ~]# docker exec -it nova_libvirt bash
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified



On container:
()[root@compute-0 /]# cat /etc/machine-id 
()[root@compute-0 /]#
Comment 3 Alexander Chuzhoy 2017-06-22 11:37:00 EDT
Was able to w/a this issue by placing a string in /etc/machine-id on the nova_compute container. It was the same string taken from the compute host.
Comment 4 Alexander Chuzhoy 2017-06-22 11:48:34 EDT
openstack-base-docker:2017-06-21.5 also has an empty /etc/machine-id file.
Comment 5 Kashyap Chamarthy 2017-06-23 09:10:32 EDT
We need to first figure out what *exactly* Nova is using the `/etc/machine-id` for.

So far, we can see that Docker is creating the /etc/machine-id, looking at this bug (thanks to Ollie Walsh for the pointer):

    https://bugzilla.redhat.com/show_bug.cgi?id=1130498

Ollie Walsh says this should probably be fixed in base RHEL / CentOS images.  As the base CentOS / RHEL images have a 0 byte 'machine-id'.

---

The official systemd documentation says, the `systemd-machine-id-setup` needs to be used to initialize the 'machine-id':

https://www.freedesktop.org/software/systemd/man/systemd-machine-id-setup.html


Also, from documentation of `machine-id(5)`:

    "This ID uniquely identifies the host. It should be considered
    "confidential", and must not be exposed in untrusted environments, 
    in particular on the network. If a stable unique identifier that is
    tied to the machine is needed for some application, the machine ID
    or any part of it must not be used directly. Instead the machine ID
    should be hashed with a cryptographic, keyed hash function, using a
    fixed, application-specific key. That way the ID will be properly
    unique, and derived in a constant way from the machine ID but there
    will be no way to retrieve the original machine ID from the
    application-specific one"
Comment 6 Kashyap Chamarthy 2017-06-23 10:33:04 EDT
(In reply to Kashyap Chamarthy from comment #5)
> We need to first figure out what *exactly* Nova is using the
> `/etc/machine-id` for.

So, looking at the code (nova/virt/libvirt/driver.py), these two methods _get_host_sysinfo_serial_os(), and _get_host_sysinfo_serial_auto() use /etc/machine-id.  Nova uses it (/etc/machine-id) for the 'sysinfo_serial' config attribute to the populate the host "serial" 'UUID exposed to guest in the virtual BIOS. Permitted options are "hardware", "os", "none" or "auto" (default):

[...]

    cfg.StrOpt('sysinfo_serial',
               default='auto',
               choices=('none', 'os', 'hardware', 'auto'),
               help='The data source used to the populate the host "serial" '
                    'UUID exposed to guest in the virtual BIOS.'),

[...]
Comment 7 Omri Hochman 2017-06-23 10:52:28 EDT
do we need to mount this file from the compute-node to the container?
Comment 8 Omri Hochman 2017-06-23 10:53:44 EDT
taking it back to container DFG, as the solution might be to remove this file from the container during prep , and it would recreate with the right content.
Comment 9 Alexander Chuzhoy 2017-08-31 12:16:56 EDT
The issue doesn't reproduce and I'm able to launch instances although the machine-id file is empty on container:

[root@overcloud-compute-0 ~]# cat /etc/machine-id 
f18324a2198a4534bb27b9d3af207b16


[root@overcloud-compute-0 ~]# docker exec -u root -it nova_libvirt bash
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
()[root@overcloud-compute-0 /]# cat /etc/machine-id 
()[root@overcloud-compute-0 /]# 


Closing the bug.

Note You need to log in before you can comment on or make changes to this bug.