Bug 1569180 - Cannot deploy OSP13 with custom hostnames - Failed running docker-puppet.py for ceilometer
Summary: Cannot deploy OSP13 with custom hostnames - Failed running docker-puppet.py f...
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 13.0 (Queens)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: 13.0 (Queens)
Assignee: Emilien Macchi
QA Contact: Gurenko Alex
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-18 18:02 UTC by Artem Hrechanychenko
Modified: 2018-10-24 15:11 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-24 15:11:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
stack failures list long (487.51 KB, text/plain)
2018-04-18 18:02 UTC, Artem Hrechanychenko
no flags Details
docker-puppet.py from failed nodes (8.62 KB, application/x-xz)
2018-04-18 18:40 UTC, Artem Hrechanychenko
no flags Details
Failures (273.83 KB, text/plain)
2018-09-25 11:07 UTC, Jaison Raju
no flags Details

Description Artem Hrechanychenko 2018-04-18 18:02:04 UTC
Created attachment 1423690 [details]
stack failures list long

Description of problem:
Cannot deploy OSP13 with 1 controller and 13 computes and custom hostnames

(undercloud) [stack@undercloud-0 ~]$ cat virt/custom_hostnames.yaml 
parameter_defaults:
  HostnameMap:
    controller-0: ctrl-0
    compute-0: test_compute-00
    compute-1: test_compute-01
    compute-2: test_compute-02
    compute-3: test_compute-03
    compute-4: test_compute-04
    compute-5: test_compute-05
    compute-6: test_compute-06
    compute-7: test_compute-07
    compute-8: test_compute-08
    compute-9: test_compute-09
    compute-10: test_compute-10
    compute-11: test_compute-11
    compute-12: test_compute-12

(undercloud) [stack@undercloud-0 ~]$ cat overcloud_deploy.sh 
#!/bin/bash

openstack overcloud deploy \
--timeout 100 \
--templates /usr/share/openstack-tripleo-heat-templates \
--stack overcloud \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /home/stack/virt/config_lvm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/custom_hostnames.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /home/stack/virt/docker-images.yaml \
--log-file overcloud_deployment_26.log

(undercloud) [stack@undercloud-0 ~]$ ironic node-list
The "ironic" CLI is deprecated and will be removed in the S* release. Please use the "openstack baremetal" CLI instead.
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 192.168.24.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 192.168.24.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
+--------------------------------------+--------------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name         | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+--------------+--------------------------------------+-------------+--------------------+-------------+
| 6d14a163-6188-4651-be31-a5946a39bd39 | compute-0    | 414d24be-d833-4369-a18c-299085efe499 | power on    | active             | False       |
| a1412bc1-5cc8-45ba-a48b-76b034672327 | compute-1    | fd2a35c6-eba6-4ddf-b055-c102e08cebf7 | power on    | active             | False       |
| 3645103e-2a0a-4e2d-a3c7-a8617414d248 | compute-10   | 3f9eafd0-9e9a-4740-9c93-615cad7ac879 | power on    | active             | False       |
| 243a05c1-e6be-4036-8789-f9296775b7ac | compute-11   | b7122211-68e0-48db-b250-47f9f440a87b | power on    | active             | False       |
| eb8b1db4-f3aa-4fdc-b6ae-0048ff582f27 | compute-12   | 85ae8ecc-0c2c-477b-a63d-ab9645ecfb4f | power on    | active             | False       |
| d81b743f-f620-4c72-a0b3-fdd17adc33f0 | compute-2    | 12ce919a-1903-4983-a712-a5acc955e395 | power on    | active             | False       |
| 74985c9b-7057-436e-b2f5-2011c367ed65 | compute-3    | 7e501818-daed-4771-be72-4bcfbb5e13f7 | power on    | active             | False       |
| 8ff208f8-b91e-4220-8f57-9ea86f21c483 | compute-4    | e22fec0f-bf59-43da-a14a-bd4ff53ae5e3 | power on    | active             | False       |
| 90e78973-fa17-421c-b1ee-f7be1dbd967b | compute-5    | 2e88847b-8937-4d04-847d-66f3be823c43 | power on    | active             | False       |
| a1faca21-e271-4224-97d8-68b7e3e4cc32 | compute-6    | 205e447e-465b-46aa-84ee-5cff1ab5fbd6 | power on    | active             | False       |
| b01648d8-3510-4924-b601-aec172d429b4 | compute-7    | 2e3d6656-dcd3-4f88-9166-00e8b69db99b | power on    | active             | False       |
| 5c518d12-12f6-47c7-943d-07f366e4ac1c | compute-8    | 6a084974-939e-4f21-b34b-37cb0e9760c7 | power on    | active             | False       |
| 02ced649-afcb-4acf-b7d3-4529843fd1c8 | compute-9    | 3795f5db-5c70-4749-bd75-3d58654a8bfe | power on    | active             | False       |
| 7d39ff0a-3806-491d-88d1-663cf0d6dca8 | controller-0 | ae13333a-fe51-4c80-9348-3d2c8d90ed73 | power on    | active             | False       |
+--------------------------------------+--------------+--------------------------------------+-------------+--------------------+-------------+

  TASK [Debug output for task which failed: Run docker-puppet tasks (generate config) during step 1] ***
    fatal: [localhost]: FAILED! => {
        "changed": false, 
        "failed_when_result": true, 
        "outputs.stdout_lines|default([])|union(outputs.stderr_lines|default([]))": [
            "2018-04-18 17:39:54,970 INFO: 17092 -- Running docker-puppet", 
            "2018-04-18 17:39:54,971 INFO: 17092 -- Service compilation completed.", 
            "2018-04-18 17:39:54,971 INFO: 17092 -- Starting multiprocess configuration steps.  Using 3 processes.", 
            "2018-04-18 17:39:55,018 INFO: 17094 -- Starting configuration of nova_libvirt using image 192.168.24.1:8787/rhosp13/openstack-nova-compute:2018-04-10.2", 
            "2018-04-18 17:39:55,018 INFO: 17093 -- Starting configuration of ceilometer using image 192.168.24.1:8787/rhosp13/openstack-ceilometer-central:2018-04-10.2", 
            "2018-04-18 17:39:55,019 INFO: 17095 -- Starting configuration of crond using image 192.168.24.1:8787/rhosp13/openstack-cron:2018-04-10.2", 
            "2018-04-18 17:39:55,020 INFO: 17094 -- Removing container: docker-puppet-nova_libvirt", 
            "2018-04-18 17:39:55,020 INFO: 17093 -- Removing container: docker-puppet-ceilometer", 
            "2018-04-18 17:39:55,028 INFO: 17095 -- Removing container: docker-puppet-crond", 
            "2018-04-18 17:39:55,103 INFO: 17093 -- Pulling image: 192.168.24.1:8787/rhosp13/openstack-ceilometer-central:2018-04-10.2", 
            "2018-04-18 17:39:55,108 INFO: 17095 -- Pulling image: 192.168.24.1:8787/rhosp13/openstack-cron:2018-04-10.2", 
            "2018-04-18 17:39:55,111 INFO: 17094 -- Pulling image: 192.168.24.1:8787/rhosp13/openstack-nova-compute:2018-04-10.2", 
            "2018-04-18 17:40:49,869 INFO: 17095 -- Removing container: docker-puppet-crond", 
            "2018-04-18 17:40:50,107 INFO: 17095 -- Finished processing puppet configs for crond", 
            "2018-04-18 17:40:50,108 INFO: 17095 -- Starting configuration of neutron using image 192.168.24.1:8787/rhosp13/openstack-neutron-server:2018-04-10.2", 
            "2018-04-18 17:40:50,125 INFO: 17095 -- Removing container: docker-puppet-neutron", 
            "2018-04-18 17:40:50,954 INFO: 17095 -- Pulling image: 192.168.24.1:8787/rhosp13/openstack-neutron-server:2018-04-10.2", 
            "2018-04-18 17:41:16,214 ERROR: 17093 -- Failed running docker-puppet.py for ceilometer", 
            "2018-04-18 17:41:16,214 ERROR: 17093 -- /usr/bin/docker-current: Error response from daemon: transport is closing.", 
            "", 
            "2018-04-18 17:41:16,214 INFO: 17093 -- Finished processing puppet configs for ceilometer", 
            "2018-04-18 17:41:16,215 INFO: 17093 -- Starting configuration of iscsid using image 192.168.24.1:8787/rhosp13/openstack-iscsid:2018-04-10.2", 
            "2018-04-18 17:41:16,227 INFO: 17093 -- Removing container: docker-puppet-iscsid", 
            "2018-04-18 17:41:16,338 INFO: 17093 -- Pulling image: 192.168.24.1:8787/rhosp13/openstack-iscsid:2018-04-10.2", 
            "2018-04-18 17:42:24,685 INFO: 17093 -- Removing container: docker-puppet-iscsid", 
            "2018-04-18 17:42:25,101 INFO: 17093 -- Finished processing puppet configs for iscsid", 
            "2018-04-18 17:42:51,947 INFO: 17095 -- Removing container: docker-puppet-neutron", 
            "2018-04-18 17:42:52,030 INFO: 17095 -- Finished processing puppet configs for neutron", 
            "2018-04-18 17:43:21,843 INFO: 17094 -- Removing container: docker-puppet-nova_libvirt", 
            "2018-04-18 17:43:21,915 INFO: 17094 -- Finished processing puppet configs for nova_libvirt", 
            "2018-04-18 17:43:21,917 ERROR: 17092 -- ERROR configuring ceilometer"
        ]
    }
    	to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/64c08a67-374d-4d86-954b-2de4383623bf_playbook.retry
    
    PLAY RECAP *********************************************************************
    localhost                  : ok=19   changed=12   unreachable=0    failed=1   
    
  deploy_stderr: |

Version-Release number of selected component (if applicable):
OSP13 with puddle 2018-04-10

How reproducible:
ALWAYS

Steps to Reproduce:
1.create custom_hostnames.yaml
parameter_defaults:
  HostnameMap:
    controller-0: ctrl-0
    compute-0: test_compute-00
    compute-1: test_compute-01
    compute-2: test_compute-02
    compute-3: test_compute-03
    compute-4: test_compute-04
    compute-5: test_compute-05
    compute-6: test_compute-06
    compute-7: test_compute-07
    compute-8: test_compute-08
    compute-9: test_compute-09
    compute-10: test_compute-10
    compute-11: test_compute-11
    compute-12: test_compute-12

2.
Deploy OSP13 using infrared 
 infrared tripleo-overcloud -v --introspect yes --containers yes --tagging yes --post no --deployment-files virt --version 13 --deploy yes --build passed_phase2 --custom-hostnames /home/ahre/infrared/custom_hostnames.yaml --specific-node-ids yes


Actual results:
Overcloud deployment were fail

Expected results:
Overcloud deployment were pass

Additional info:

Comment 3 Emilien Macchi 2018-04-18 18:12:19 UTC
"2018-04-18 17:41:14,383 ERROR: 17076 -- /usr/bin/docker-current: Error response from daemon: transport is closing.", 

Not sure it's related to custom hostnames, but the logs provided don't show any other trace.

Could you please run "export DEBUG=true && python /var/lib/docker-puppet/docker-puppet.py" and show the output.

Thanks

Comment 4 Artem Hrechanychenko 2018-04-18 18:40:35 UTC
Created attachment 1423706 [details]
docker-puppet.py from failed nodes

Comment 8 Jaison Raju 2018-09-25 11:07:38 UTC
Created attachment 1486717 [details]
Failures

Comment 10 Alex Schultz 2018-10-24 15:11:30 UTC
The provided debug log did not have the error in it. Closing for now, feel free to reopen with additional logs if this is still an issue.


Note You need to log in before you can comment on or make changes to this bug.