Bug 1480172 - OSP11 -> OSP12 upgrade: during upgrade the switch-from-non-containerized-to-containerized-ceph-daemons.yml provided by ceph-ansible fails because it requires user input
OSP11 -> OSP12 upgrade: during upgrade the switch-from-non-containerized-to-c...
Status: VERIFIED
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
urgent Severity urgent
: rc
: 12.0 (Pike)
Assigned To: Giulio Fidente
Yogev Rabl
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-10 06:23 EDT by Marius Cornea
Modified: 2017-11-15 22:35 EST (History)
20 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-7.0.0-0.20170821194253.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 494246 None None None 2017-08-16 11:41 EDT
Github ceph/ceph-ansible/issues/1758 None None None 2017-08-10 07:38 EDT

  None (edit)
Description Marius Cornea 2017-08-10 06:23:35 EDT
Description of problem:
OSP11 -> OSP12 upgrade: major-upgrade-composable-steps-docker fails with resources.WorkflowTasks_Step2_Execution: ERROR when upgrading ceph deployments with ceph-ansible.yaml

Version-Release number of selected component (if applicable):
ceph-ansible-2.3.0-0.1.rc5.el7cp.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170805163045.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy basic OSP11 environment with 1 ctrl, 1 compute, 1 ceph node:

timeout 100m openstack overcloud deploy \
--templates /usr/share/openstack-tripleo-heat-templates \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \
-e /home/stack/virt/internal.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml 

2. Upgrade undercloud to OSP12

3. Run major-upgrade-composable-steps-docker upgrade step and switch to ceph-ansible:

timeout 100m openstack overcloud deploy \
--templates /usr/share/openstack-tripleo-heat-templates \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
-e /home/stack/virt/internal.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps-docker.yaml \
-e /home/stack/docker-osp12.yaml \

Set the CephAnsiblePlaybook parameter to point to the switch-from-non-containerized-to-containerized-ceph-daemons.yml playbook:

CephAnsiblePlaybook: /home/stack/switch-from-non-containerized-to-containerized-ceph-daemons.yml

Actual results:
major-upgrade-composable-steps-docker fails

Expected results:
upgrade completes fine.

Additional info:

The ceph-ansible command:

', u'become': True, u'forks': 3, u'ssh_common_args': u'-o StrictHostKeyChecking=no', u'playbook': u'/home/stack/switch-from-non-containerized-to-containerized-ceph-daemons.yml'}']                                
 Unexpected error while running command.                                                                                                                                                                           
Command: ansible-playbook /tmp/ansible-mistral-actiono9ievc/playbook.yaml --user tripleo-admin --become --become-user root --extra-vars {\"monitor_secret\": \"***\", \"ceph_conf_overrides\": {\"global\": {\"osd_
pool_default_pg_num\": 32, \"osd_pool_default_size\": 1}}, \"fetch_directory\": \"/tmp/file-mistral-actionWNAiYg\", \"user_config\": true, \"ceph_docker_image_tag\": \"latest\", \"containerized_deployment\": tru
e, \"public_network\": \"172.17.3.0/24\", \"generate_fsid\": false, \"monitor_address_block\": \"172.17.3.0/24\", \"monitor_interface\": \"br_ex\", \"admin_secret\": \"***\", \"keys\": [{\"mon_cap\": \"allow r\"
, \"osd_cap\": \"allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics\", \"acls\": [\"u:glance:r--\", \"u
:nova:r--\", \"u:cinder:r--\", \"u:gnocchi:r--\"], \"name\": \"client.openstack\", \"key\": \"AQAuZ4tZAAAAABAAurr1kyUdEM3jwPepTiOrSQ==\"}], \"openstack_keys\": [{\"mon_cap\": \"allow r\", \"osd_cap\": \"allow cl
ass-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics\", \"acls\": [\"u:glance:r--\", \"u:nova:r--\", \"u:cinder:r
--\", \"u:gnocchi:r--\"], \"name\": \"client.openstack\", \"key\": \"AQAuZ4tZAAAAABAAurr1kyUdEM3jwPepTiOrSQ==\"}], \"journal_collocation\": true, \"ntp_service_enabled\": false, \"ceph_docker_image\": \"rhceph-2
-rhel7\", \"cluster_network\": \"172.17.4.0/24\", \"fsid\": \"cb721718-7d3b-11e7-ad93-5254008ebd12\", \"journal_size\": 512, \"openstack_config\": true, \"ceph_docker_registry\": \"192.168.24.1:8787/ceph\", \"ce
ph_stable\": true, \"devices\": [\"/dev/vdb\"], \"ceph_origin\": \"distro\", \"openstack_pools\": [{\"rule_name\": \"\", \"pg_num\": 32, \"name\": \"volumes\"}, {\"rule_name\": \"\", \"pg_num\": 32, \"name\": \"
backups\"}, {\"rule_name\": \"\", \"pg_num\": 32, \"name\": \"vms\"}, {\"rule_name\": \"\", \"pg_num\": 32, \"name\": \"images\"}, {\"rule_name\": \"\", \"pg_num\": 32, \"name\": \"metrics\"}], \"pools\": [], \"
docker\": true, \"ip_version\": \"ipv4\"} --forks 3 --ssh-common-args \"-o StrictHostKeyChecking=no\" --ssh-extra-args \"-o UserKnownHostsFile=/dev/null\" --inventory-file /tmp/ansible-mistral-actiono9ievc/inven
tory.yaml --private-key /tmp/ansible-mistral-actiono9ievc/ssh_private_key --skip-tags package-install,with_pkg                                                                                                     
Exit code: 4                                                                                                                                                                                                       
Stdout: u''                                                                                                                                                                                                        
Stderr: u\"[WARNING]: log file at /var/log/ansible.log is not writeable and we cannot create it, aborting\                                                                                                         
\                                                                                                                                                                                                                  
ERROR! playbooks must be a list of plays\                                                                                                                                                                          
\                                                                                                                                                                                                                  
The error appears to have been in '/tmp/ansible-mistral-actiono9ievc/playbook.yaml': line 1, column 1, but may\                                                                                                    
be elsewhere in the file depending on the exact syntax problem.\                                                                                                                                                   
\                                                                                                                                                                                                                  
The offending line appears to be:\                                                                                                                                                                                 
\                                                                                                                                                                                                                  
\                                                                                                                                                                                                                  
/home/stack/switch-from-non-containerized-to-containerized-ceph-daemons.yml\                                                                                                                                       
^ here\                                                                                                                                                                                                            
\""                                                                                                                                                                                                                


The inventory file:
inventory.yaml:
clients:                      
  hosts:                      
    192.168.24.9: {}          
mdss:                         
  hosts: {}                   
mons:                         
  hosts:                      
    192.168.24.17: {}         
osds:                         
  hosts:                      
    192.168.24.13: {}         
rgws:                         
  hosts: {}
Comment 1 Giulio Fidente 2017-08-10 06:53:51 EDT
Content of the playbook.yaml file is as follows:

# more playbook.yaml 
/home/stack/switch-from-non-containerized-to-containerized-ceph-daemons.yml
Comment 2 Giulio Fidente 2017-08-10 06:58:27 EDT
Looks like the playbook to run is misintepreted as a string to build the playbook from instead of copying the contents of the file into playbook.yaml
Comment 3 Giulio Fidente 2017-08-10 07:11:20 EDT
I believe the issue is that /home/stack/switch-from-non-containerized-to-containerized-ceph-daemons.yml was not readable by mistral. 

If we set CephAnsiblePlaybook: to /usr/share/ceph-ansible/infrastructure-playbooks/switch-from-non-containerized-to-containerized-ceph-daemons.yml the playbook should run (and probably fail later on something different).

Can you confirm the above relaunching the upgrade?
Comment 4 Marius Cornea 2017-08-10 07:36:12 EDT
(In reply to Giulio Fidente from comment #3)
> I believe the issue is that
> /home/stack/switch-from-non-containerized-to-containerized-ceph-daemons.yml
> was not readable by mistral. 
> 
> If we set CephAnsiblePlaybook: to
> /usr/share/ceph-ansible/infrastructure-playbooks/switch-from-non-
> containerized-to-containerized-ceph-daemons.yml the playbook should run (and
> probably fail later on something different).
> 
> Can you confirm the above relaunching the upgrade?

So we changed CephAnsiblePlaybook to point to the one provided in /usr/share and now it failed as the playbook requires user input:

https://github.com/ceph/ceph-ansible/blob/master/infrastructure-playbooks/switch-from-non-containerized-to-containerized-ceph-daemons.yml#L11-L15
Comment 7 Giulio Fidente 2017-08-11 02:55:58 EDT
We can workaround the prompt passing its value via THT
Comment 12 Yogev Rabl 2017-11-15 22:35:05 EST
verified on openstack-tripleo-heat-templates-7.0.3-6.el7ost.noarch

Note You need to log in before you can comment on or make changes to this bug.