Created attachment 1423067 [details] ceph sosreport Description of problem: After updating osp12 GA to latest (2018-04-04) tempest test fails to create volume Version-Release number of selected component (if applicable): python-cinder-11.1.0-1.el7ost.noarch python-cinderclient-3.1.0-1.el7ost.noarch openstack-cinder-11.1.0-1.el7ost.noarch puppet-cinder-11.5.0-3.el7ost.noarch eph-selinux-10.2.10-17.el7cp.x86_64 ceph-mon-10.2.10-17.el7cp.x86_64 puppet-ceph-2.4.2-1.el7ost.noarch collectd-ceph-5.7.2-1.1.el7ost.x86_64 ceph-radosgw-10.2.10-17.el7cp.x86_64 libcephfs1-10.2.10-17.el7cp.x86_64 ceph-common-10.2.10-17.el7cp.x86_64 python-cephfs-10.2.10-17.el7cp.x86_64 ceph-mds-10.2.10-17.el7cp.x86_64 ceph-base-10.2.10-17.el7cp.x86_64 How reproducible: Steps to Reproduce: 1. Install osp12 (rhel 7.4) 2. update cloud to latest with rhel 7.5 (2018-04-04) 3. Run tempest 'sanity' test Actual results: Expected results: Additional info:
Created attachment 1423071 [details] tempest results
Created attachment 1423072 [details] tempest results (xml)
This problem occurred before rebooting the ceph nodes. After rebooting ceph nodes tempest tests pass.
Wrong component? Why is this a bug on ceph-ansible? Can we get some clarification on this? Thanks.
Raviv, are you able to reproduce this bug consistently? My understanding is that on an attempt to upgrade rhel using director, the ceph update (which is running site-docker.yml.sample against the cluster created on the initial deployment) succeeded *but* cinder was unable to create new volumes. Is this correct? If so, can you attach the /var/log/mistral logs from the undercloud? Yogev, are we able to reproduce this test?
It might also be useful to collect cinder-volume logs, which probably have hints about why volume creation is failing... cause this probably is not a ceph-ansible issue.
Created attachment 1478797 [details] ceph logs
Created attachment 1478798 [details] compute logs
This bug is reproducible, the automatic job reproducing the bug is: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-upgrades-updates-12_director-rhel-7.4-virthost-3cont_2comp_3ceph-ipv4-vxlan-os-7.5 I attached controller log for more info and additional logs can be extracted from the job itself https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-upgrades-updates-12_director-rhel-7.4-virthost-3cont_2comp_3ceph-ipv4-vxlan-os-7.5/27//artifact/tempest-results/
Hi Giulio, You can check this CI job: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/upgrades/view/update/job/DFG-upgrades-updates-12_director-rhel-7.4-virthost-3cont_2comp_3ceph-ipv4-vxlan-os-7.5/28/artifact/