Bug 1519055

I checked the environment and the failure was caused by an incorrect tag specified for the Docker images during upgrade. We're currently trying to run the upgrade with the correct tag:

Nov 29 21:51:57 monitor-0 os-collect-config[2955]: "Pulling repository docker-registry.engineering.redhat.com/rhosp12/openstack-cron-docker\", \n        \"/usr/bin/docker-current: Error: image rhosp12/openstack-cron-docker:20171128.1 not found.\", \n        \"See '/usr/bin/docker-current run --help'.\", \n        \"2017-11-29 21:51:56,898 INFO: 54294 -- Finished processing puppet configs\", \n        \"2017-11-29 21:51:56,899 ERROR: 54293 -- ERROR configuring crond\"\n    ], \n    \"failed_when_result\": true\n}\n\tto retry, use: --limit @/var/lib/heat-config/heat-config-ansible/a245485f-74a1-45eb-b430-54a6e480a25b_playbook.retry\n\nPLAY RECAP *********************************************************************\nlocalhost                  : ok=6    changed=2    unreachable=0    failed=1   \n\n", "deploy_stderr": "", "deploy_status_code": 2}

After re-run ceph-ansible fails with:

2017-11-30 11:35:33,655 p=28327 u=mistral |  skipping: [192.168.24.20]
2017-11-30 11:35:33,665 p=28327 u=mistral |  TASK [ceph-mon : populate kv_store with custom ceph.conf] **********************
2017-11-30 11:35:33,686 p=28327 u=mistral |   [WARNING]: when statements should not include jinja2 templating delimiters
such as {{ }} or {% %}. Found: {{ inventory_hostname ==
groups[mon_group_name][0] }}

2017-11-30 11:35:33,694 p=28327 u=mistral |  skipping: [192.168.24.20]
2017-11-30 11:35:33,706 p=28327 u=mistral |  TASK [ceph-mon : delete populate-kv-store docker] ******************************
2017-11-30 11:35:33,726 p=28327 u=mistral |  skipping: [192.168.24.20]
2017-11-30 11:35:33,737 p=28327 u=mistral |  TASK [ceph-mon : generate systemd unit file] ***********************************
2017-11-30 11:35:33,849 p=28327 u=mistral |  fatal: [192.168.24.20]: FAILED! => {"changed": false, "failed": true, "msg": "AnsibleUndefinedVariable: No first item, sequence was empty."}
2017-11-30 11:35:33,851 p=28327 u=mistral |  PLAY RECAP *********************************************************************
2017-11-30 11:35:33,851 p=28327 u=mistral |  192.168.24.11              : ok=4    changed=0    unreachable=0    failed=0   
2017-11-30 11:35:33,851 p=28327 u=mistral |  192.168.24.13              : ok=4    changed=0    unreachable=0    failed=0   
2017-11-30 11:35:33,852 p=28327 u=mistral |  192.168.24.15              : ok=4    changed=0    unreachable=0    failed=0   
2017-11-30 11:35:33,852 p=28327 u=mistral |  192.168.24.20              : ok=40   changed=4    unreachable=0    failed=1   
2017-11-30 11:35:33,852 p=28327 u=mistral |  192.168.24.6               : ok=4    changed=0    unreachable=0    failed=0   
2017-11-30 11:35:33,852 p=28327 u=mistral |  192.168.24.7               : ok=4    changed=0    unreachable=0    failed=0   
2017-11-30 11:35:33,852 p=28327 u=mistral |  localhost                  : ok=0    changed=0    unreachable=0    failed=0

This is the ansible-playbook command:

2017-11-30 11:35:33.956 1406 ERROR mistral.executors.default_executor Command: ansible-playbook /usr/share/ceph-ansible/infrastructure-playbooks/switch-from-non-containerized-to-containerized-ceph-daemons.yml --user tripleo-admin --become 
--become-user root --extra-vars {"monitor_secret": "***", "ceph_conf_overrides": {"global": {"rgw_s3_auth_use_keystone": "true", "rgw_keystone_admin_password": "***", "osd_pool_default_pgp_num": 128, "rgw_keystone_url": "http://172.17.1.13
:5000", "rgw_keystone_admin_project": "service", "rgw_keystone_accepted_roles": "Member, _member_, admin", "osd_pool_default_size": 3, "osd_pool_default_pg_num": 128, "rgw_keystone_api_version": 3, "rgw_keystone_admin_user": "swift", "rgw_
keystone_admin_domain": "default"}}, "fetch_directory": "/tmp/file-mistral-actionX1Z8i0", "user_config": true, "ceph_docker_image_tag": "latest", "ceph_release": "jewel", "containerized_deployment": true, "public_network": "172.17.3.0/24",
 "generate_fsid": false, "monitor_address_block": "172.17.3.0/24", "admin_secret": "***", "keys": [{"mon_cap": "allow r", "osd_cap": "allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx po
ol=vms, allow rwx pool=images, allow rwx pool=metrics", "name": "client.openstack", "key": "AQDJCh9aAAAAABAAl6AovurIbHqTpfsW8rzprg==", "mode": "0644"}, {"mon_cap": "allow r, allow command \\\\\\\"auth del\\\\\\\", allow command \\\\\\\"aut
h caps\\\\\\\", allow command \\\\\\\"auth get\\\\\\\", allow command \\\\\\\"auth get-or-create\\\\\\\"", "mds_cap": "allow *", "name": "client.manila", "mode": "0644", "key": "AQDJCh9aAAAAABAA+T296HHhVCoOYsPzS1vNmg==", "osd_cap": "allow 
rw"}, {"mon_cap": "allow rw", "osd_cap": "allow rwx", "name": "client.radosgw", "key": "AQDJCh9aAAAAABAAxn7AQ33TrNbc6WHUc3YdlQ==", "mode": "0644"}], "openstack_keys": [{"mon_cap": "allow r", "osd_cap": "allow class-read object_prefix rbd_c
hildren, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics", "name": "client.openstack", "key": "AQDJCh9aAAAAABAAl6AovurIbHqTpfsW8rzprg==", "mode": "0644"}, {"mon_cap": "allow
 r, allow command \\\\\\\"auth del\\\\\\\", allow command \\\\\\\"auth caps\\\\\\\", allow command \\\\\\\"auth get\\\\\\\", allow command \\\\\\\"auth get-or-create\\\\\\\"", "mds_cap": "allow *", "name": "client.manila", "mode": "0644", 
"key": "AQDJCh9aAAAAABAA+T296HHhVCoOYsPzS1vNmg==", "osd_cap": "allow rw"}, {"mon_cap": "allow rw", "osd_cap": "allow rwx", "name": "client.radosgw", "key": "AQDJCh9aAAAAABAAxn7AQ33TrNbc6WHUc3YdlQ==", "mode": "0644"}], "osd_objectstore": "f
ilestore", "pools": [], "ntp_service_enabled": false, "ceph_docker_image": "ceph/rhceph-2-rhel7", "cluster_network": "172.17.4.0/24", "fsid": "bb349fe4-d53b-11e7-9601-525400f22762", "openstack_config": true, "ceph_docker_registry": "rhos-q
e-mirror-tlv.usersys.redhat.com:5000", "ceph_stable": true, "devices": ["/dev/vdb"], "ceph_origin": "distro", "openstack_pools": [{"rule_name": "", "pg_num": 128, "name": "volumes"}, {"rule_name": "", "pg_num": 128, "name": "backups"}, {"r
ule_name": "", "pg_num": 128, "name": "vms"}, {"rule_name": "", "pg_num": 128, "name": "images"}, {"rule_name": "", "pg_num": 128, "name": "metrics"}], "ip_version": "ipv4", "ireallymeanit": "yes", "docker": true} --forks 11 --inventory-fi
le /tmp/ansible-mistral-actionROSLKc/inventory.yaml --private-key /tmp/ansible-mistral-actionROSLKc/ssh_private_key --skip-tags package-install,with_pkg

Inventory:

id=eab906f8-17fa-4bd5-ad9d-c00b6a94d2da, action_cls_str=tripleo_common.actions.ansible.AnsiblePlaybookAction, action_cls_attrs={}, params={remote_user: tripleo-admin,
inventory: {u'mdss': {u'hosts': {}}, u'rgws': {u'hosts': {}}, u'mons': {                                                                                              
u'hosts': {u'192.168.24.20': {}, u'192.168.24.7': {}, u'192.168.24.6': {}}}, u'clients': {u'hosts': {u'192.168.24.17': {}, u'192.168.24.18': {}, u'192.168.24.9': {}, 
u'192.168.24.8': {}, u'192.168.24.20': {}, u'192.168.24.23': {}, u'192.16                                                                                             
8.24.7': {}, u'192.168.24.6': {}}}, u'osds': {u'hosts': {u'192.168.24.11': {}, u'192.168.24.13': {}, u'192.168.24.15': {}}}}...]

Created attachment 1361029 [details]
ceph-install-workflow.log

Probably a duplicate of BZ #1519052

Hi Yogev,

I think the error which showed up during the upgrade was caused by the incorrect network configuration during OSP11 deployment that I was mentioning last week. I tried a quick test with this Infrared patch[1] applied and upgrade completed fine for me. Could you please check if it fixes the issue for you?

Thanks,
Marius

[1] https://review.gerrithub.io/#/c/389884/

(In reply to Marius Cornea from comment #15)
> Hi Yogev,
> 
> I think the error which showed up during the upgrade was caused by the
> incorrect network configuration during OSP11 deployment that I was
> mentioning last week. I tried a quick test with this Infrared patch[1]
> applied and upgrade completed fine for me. Could you please check if it
> fixes the issue for you?
> 
> Thanks,
> Marius
> 
> [1] https://review.gerrithub.io/#/c/389884/

Tested with you fix and it worked. thanks!