Bug 1480326 - openstack-nova: failed to launch instance on OC: "Host 'overcloud-compute-1.localdomain' is not mapped to any cell", "code": 400, "created": "2017-08-10T13:16:11Z"}
openstack-nova: failed to launch instance on OC: "Host 'overcloud-compute-1.l...
Status: CLOSED WORKSFORME
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
unspecified Severity unspecified
: rc
: 12.0 (Pike)
Assigned To: Eoghan Glynn
Joe H. Rahme
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-10 13:27 EDT by Alexander Chuzhoy
Modified: 2017-09-08 05:46 EDT (History)
15 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-09-08 05:45:42 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
mistral logs from UC. (1.05 MB, application/x-gzip)
2017-08-10 13:27 EDT, Alexander Chuzhoy
no flags Details
nova logs from controller2 (1.66 MB, application/x-gzip)
2017-08-22 16:17 EDT, Alexander Chuzhoy
no flags Details
nova logs from controller1 (1.65 MB, application/x-gzip)
2017-08-22 16:18 EDT, Alexander Chuzhoy
no flags Details
nova logs from controller0 (1.70 MB, application/x-gzip)
2017-08-22 16:19 EDT, Alexander Chuzhoy
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 497955 None None None 2017-08-25 11:17 EDT

  None (edit)
Description Alexander Chuzhoy 2017-08-10 13:27:07 EDT
openstack-nova: failed to launch instance on OC: "Host 'overcloud-compute-1.localdomain' is not mapped to any cell", "code": 400, "created": "2017-08-10T13:16:11Z"}

Environment:
openstack-tripleo-heat-templates-7.0.0-0.20170805163045.el7ost.noarch
instack-undercloud-7.2.1-0.20170729010705.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch

libvirt-daemon-driver-storage-rbd-3.2.0-14.el7.x86_64
openstack-nova-scheduler-16.0.0-0.20170805120344.5971dde.el7ost.noarch
libvirt-daemon-driver-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-lxc-3.2.0-14.el7.x86_64
libvirt-daemon-driver-qemu-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7.x86_64
python-nova-16.0.0-0.20170805120344.5971dde.el7ost.noarch
openstack-nova-console-16.0.0-0.20170805120344.5971dde.el7ost.noarch
libvirt-libs-3.2.0-14.el7.x86_64
libvirt-daemon-driver-interface-3.2.0-14.el7.x86_64
libvirt-client-3.2.0-14.el7.x86_64
libvirt-daemon-driver-nodedev-3.2.0-14.el7.x86_64
puppet-nova-11.3.0-0.20170805105252.30a205c.el7ost.noarch
libvirt-daemon-driver-storage-core-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-logical-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-disk-3.2.0-14.el7.x86_64
openstack-nova-conductor-16.0.0-0.20170805120344.5971dde.el7ost.noarch
openstack-nova-novncproxy-16.0.0-0.20170805120344.5971dde.el7ost.noarch
libvirt-daemon-driver-network-3.2.0-14.el7.x86_64
libvirt-python-3.2.0-3.el7.x86_64
libvirt-daemon-config-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-gluster-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-3.2.0-14.el7.x86_64
openstack-nova-compute-16.0.0-0.20170805120344.5971dde.el7ost.noarch
openstack-nova-migration-16.0.0-0.20170805120344.5971dde.el7ost.noarch
openstack-nova-api-16.0.0-0.20170805120344.5971dde.el7ost.noarch
libvirt-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-scsi-3.2.0-14.el7.x86_64
libvirt-daemon-kvm-3.2.0-14.el7.x86_64
openstack-nova-common-16.0.0-0.20170805120344.5971dde.el7ost.noarch
libvirt-daemon-3.2.0-14.el7.x86_64
libvirt-daemon-driver-secret-3.2.0-14.el7.x86_64
python-novaclient-9.1.0-0.20170804194758.0a53d19.el7ost.noarch
libvirt-daemon-driver-storage-mpath-3.2.0-14.el7.x86_64
openstack-nova-placement-api-16.0.0-0.20170805120344.5971dde.el7ost.noarch
libvirt-daemon-config-network-3.2.0-14.el7.x86_64


Steps to reproduce:
1. Deploy OC (used external ceph)
2. Try to launch an instance

Result:
(overcloud) [stack@undercloud-0 ~]$ nova list
+--------------------------------------+--------+--------+------------+-------------+----------+
| ID                                   | Name   | Status | Task State | Power State | Networks |
+--------------------------------------+--------+--------+------------+-------------+----------+
| 59799ce9-c4b7-48ca-9419-7a81a9f6b549 | nisim1 | ERROR  | -          | NOSTATE     |          |
+--------------------------------------+--------+--------+------------+-------------+----------+



Checking the fault:

| fault                                | {"message": "Host 'overcloud-compute-1.localdomain' is not mapped to any cell", "code": 400, "created": "2017-08-10T13:16:11Z"} |
Comment 1 Alexander Chuzhoy 2017-08-10 13:27 EDT
Created attachment 1311851 [details]
mistral logs from UC.
Comment 3 Ollie Walsh 2017-08-17 12:40:18 EDT
(In reply to Alexander Chuzhoy from comment #1)
> Created attachment 1311851 [details]
> mistral logs from UC.

Not relevant for the overcloud nodes BTW
Comment 5 Ollie Walsh 2017-08-18 09:33:46 EDT
Could you try to reproduce to get logs?
Comment 6 Alexander Chuzhoy 2017-08-21 14:36:57 EDT
Reproduced on a deployment with external ceph (only).
Collecting sosreports.
Comment 8 Lee Yarwood 2017-08-22 05:15:35 EDT
(In reply to Alexander Chuzhoy from comment #0)
> Steps to reproduce:
> 1. Deploy OC (used external ceph)
> 2. Try to launch an instance

Fun, the sosrepors don't contain the overcloud deploy log on the undercloud or any nova logs in the overcloud. Can you attach both the deploy log and /var/log/containers/nova*/nova-manage logs from each of the overcloud controllers?

For a pure containers deployment we should see the discover hosts command issued as part of the following n-api setup :

https://github.com/openstack/tripleo-heat-templates/blob/84ef6a5342b113a9807f2c5c9587178d4cbf02ef/docker/services/nova-api.yaml#L208
Comment 9 Alexander Chuzhoy 2017-08-22 16:17:02 EDT
The only line in the deployment log is:
(undercloud) [stack@undercloud-0 ~]$ cat overcloud_deployment_0.log
2017-08-21 11:34:46.817 8855 WARNING tripleoclient.plugin [  admin] Waiting for messages on queue '2719e72f-8b59-459b-9d80-7628eb756d92' with no timeout.

Adding the nova logs from controllers.
Comment 10 Alexander Chuzhoy 2017-08-22 16:17 EDT
Created attachment 1316790 [details]
nova logs from controller2
Comment 11 Alexander Chuzhoy 2017-08-22 16:18 EDT
Created attachment 1316791 [details]
nova logs from controller1
Comment 12 Alexander Chuzhoy 2017-08-22 16:19 EDT
Created attachment 1316792 [details]
nova logs from controller0
Comment 13 Ollie Walsh 2017-08-22 16:40:52 EDT
/var/log/messages from the controllers is what I'm after to see the nova-manage command output
Comment 14 Ollie Walsh 2017-08-22 16:48:19 EDT
actually I don't see any output /var/log/messages. Dan, do we log the output from the nova_api_discover_hosts container?
Comment 15 Ollie Walsh 2017-08-25 09:18:20 EDT
Dan, I think this was broken by https://review.openstack.org/465551. The bootstrap_host command fails when I run it manually:

[root@overcloud-controller-0 heat-admin]# docker run -e "TRIPLEO_CONFIG_HASH=5c47cd23408c0828a337c6ebb3302060"  -e "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"  -e "KOLLA_BASE_DISTRO=cens"  -e "KOLLA_INSTALL_TYPE=binary"  -e "KOLLA_INSTALL_METATYPE=rdo"  -e "PS1=$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ "   -v /etc/puppet:/etc/puppet:ro -v /dev/log:/dev/log -v /var/lib/config-data/nova/etc/nova:/etc/nova:ro -v /var/log/containers/nova:/var/log/nova -v /etc/hosts:/etc/hosts:ro -v /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro -v /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro -v /etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro -v /etc/localtime:/etc/localtime:ro -v /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro -v /var/lib/config-data/nova/etc/my.cnf.d/tripleo.cnf:/etc/my.cnf.d/tripleo.cnf:ro -v /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro -ti 192.168.24.1:8787/tripleoupstream/centos-binary-nova-api:latest /usr/bin/bootstrap_host_exec nova_api su nova -s /bin/bash -c '/usr/bin/nova-manage cell_v2 discover_hosts'
Failed to start Hiera: Errno::EACCES: Permission denied - /etc/puppet/hiera.yaml

And when I remove the bootstrap_host command it then fails to su:

[root@overcloud-controller-0 heat-admin]# docker run -e "TRIPLEO_CONFIG_HASH=5c47cd23408c0828a337c6ebb3302060"  -e "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"  -e "KOLLA_BASE_DISTRO=centos"  -e "KOLLA_INSTALL_TYPE=binary"  -e "KOLLA_INSTALL_METATYPE=rdo"  -e "PS1=$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ "   -v /etc/puppet:/etc/puppet:ro -v /dev/log:/dev/log -v /var/lib/config-data/nova/etc/nova:/etc/nova:ro -v /var/log/containers/nova:/var/log/nova -v /etc/hosts:/etc/hosts:ro -v /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro -v /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro -v /etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro -v /etc/localtime:/etc/localtime:ro -v /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro -v /var/lib/config-data/nova/etc/my.cnf.d/tripleo.cnf:/etc/my.cnf.d/tripleo.cnf:ro -v /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro -ti 192.168.24.1:8787/tripleoupstream/centos-binary-nova-api:latest su nova -s /bin/bash -c '/usr/bin/nova-manage cell_v2 discover_hosts'
Password: 
su: Authentication failure
Comment 16 Ollie Walsh 2017-08-25 09:52:34 EDT
Seems the instructions in https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/tips_tricks.html#debugging-container-failures are not correct in this case. When I query paunch and run the command I hit a quoting issue instead:

[root@overcloud-controller-0 heat-admin]# paunch debug --file /var/lib/tripleo-config/hashed-docker-container-startup-config-step_5.json --container nova_api_discover_hosts --action print-cmd
docker run --name nova_api_discover_hosts-567af21w --env=TRIPLEO_CONFIG_HASH=5c47cd23408c0828a337c6ebb3302060 --net=host --user=root --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/var/lib/config-data/nova/etc/my.cnf.d/tripleo.cnf:/etc/my.cnf.d/tripleo.cnf:ro --volume=/var/lib/config-data/nova/etc/nova/:/etc/nova/:ro --volume=/var/log/containers/nova:/var/log/nova 192.168.24.1:8787/tripleoupstream/centos-binary-nova-api:latest /usr/bin/bootstrap_host_exec nova_api su nova -s /bin/bash -c '/usr/bin/nova-manage cell_v2 discover_hosts'
[root@overcloud-controller-0 heat-admin]# docker run --name nova_api_discover_hosts-567af21w --env=TRIPLEO_CONFIG_HASH=5c47cd23408c0828a337c6ebb3302060 --net=host --user=root --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/var/lib/config-data/nova/etc/my.cnf.d/tripleo.cnf:/etc/my.cnf.d/tripleo.cnf:ro --volume=/var/lib/config-data/nova/etc/nova/:/etc/nova/:ro --volume=/var/log/containers/nova:/var/log/nova 192.168.24.1:8787/tripleoupstream/centos-binary-nova-api:latest /usr/bin/bootstrap_host_exec nova_api su nova -s /bin/bash -c '/usr/bin/nova-manage cell_v2 discover_hosts'
usage: nova-manage [-h] [--config-dir DIR] [--config-file PATH] [--debug]
                   [--log-config-append PATH] [--log-date-format DATE_FORMAT]
                   [--log-dir LOG_DIR] [--log-file PATH] [--nodebug]
                   [--nopost-mortem] [--nouse-journal] [--nouse-syslog]
                   [--nowatch-log-file] [--post-mortem]
                   [--syslog-log-facility SYSLOG_LOG_FACILITY] [--use-journal]
                   [--use-syslog] [--version] [--watch-log-file]
                   [--remote_debug-host REMOTE_DEBUG_HOST]
                   [--remote_debug-port REMOTE_DEBUG_PORT]
                   
                   {version,bash-completion,shell,logs,cell_v2,db,quota,agent,host,floating,api_db,project,account,network,cell}
                   ...
nova-manage: error: too few arguments

Works when I escape the quotes:
[root@overcloud-controller-0 heat-admin]# docker run --rm --name nova_api_discover_hosts-567af21w --env=TRIPLEO_CONFIG_HASH=5c47cd23408c0828a337c6ebb3302060 --net=host --user=root --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/var/lib/config-data/nova/etc/my.cnf.d/tripleo.cnf:/etc/my.cnf.d/tripleo.cnf:ro --volume=/var/lib/config-data/nova/etc/nova/:/etc/nova/:ro --volume=/var/log/containers/nova:/var/log/nova 192.168.24.1:8787/tripleoupstream/centos-binary-nova-api:latest /usr/bin/bootstrap_host_exec nova_api su nova -s /bin/bash -c \'/usr/bin/nova-manage cell_v2 discover_hosts\'
[root@overcloud-controller-0 heat-admin]#
Comment 17 Dan Prince 2017-08-25 10:36:53 EDT
(In reply to Ollie Walsh from comment #14)
> actually I don't see any output /var/log/messages. Dan, do we log the output
> from the nova_api_discover_hosts container?

I don't think we cleanup this container. I think you should still see it via 'docker ps -a' on the overcloud controller where it executes. If there is logging to stdout you should be able to obtain that via 'docker logs <container_name>'.

If the logs go to a file then those would show up in /var/log/containers/nova I think.
Comment 18 Ollie Walsh 2017-08-25 11:17:51 EDT
We have nothing. I've proposed https://review.openstack.org/497955 to run with --verbose so we get something on stdout.

If it's possible to reproduce can you try with this change?
Comment 19 Artem Hrechanychenko 2017-09-06 09:47:12 EDT
(In reply to Ollie Walsh from comment #18)
> We have nothing. I've proposed https://review.openstack.org/497955 to run
> with --verbose so we get something on stdout.
> 
> If it's possible to reproduce can you try with this change?

I'll check this
Comment 20 Alexander Chuzhoy 2017-09-06 20:32:07 EDT
Seems like the issue doesn't reproduce now.
I was able to launch instance using external ceph.
Comment 21 Ollie Walsh 2017-09-08 05:45:42 EDT
Closing this as it can't be reproduced but adding some logging to help debug any issues in future.

Note You need to log in before you can comment on or make changes to this bug.