Description of problem: After recent updates, we cannot re-attach volumes. Version-Release number of selected component (if applicable): We just run RDO (Juno) with Centos 7, and we usually wait a week or two to apply updates. How reproducible: Hard to tell. We're stuck on this situation and we cannot go back to try, as yum downgrade failed. Steps to Reproduce: 1. Run RDO. Stop instances. 2. Apply these recent updates today http://fpaste.org/204340/27597219/ 3. Reboot 4. Try to bring things up, and the attach of a local volume fails Actual results: this is the error in cinder/volume.log: http://fpaste.org/204339/14275970/ Expected results: Be able to run our instances again! Additional info: I tried to diagnose if the issue was with logging (after seeing the Oslo people report that logging has to change across libraries to use unicode), by comenting out this line 522 of /usr/lib/python2.7/site-packages/cinder/brick/iscsi/iscsi.py: LOG.info(_('Creating iscsi_target for volume: %s') % vol_id) other than that, nothing else is changed.
Created attachment 1007901 [details] yum.log
Created attachment 1007902 [details] cinder volume.log
Thanks for the report! We try to discourage people from referring to pastebins in bug reports, because when the paste expires the bug becomes less useful. I've attached your two pastes to this bug, so you're all set for now.
yum.log, without html Mar 28 13:47:31 rack1 yum[4170]: Updated: wxBase-2.8.12-13.el7.x86_64 Mar 28 13:47:32 rack1 yum[4170]: Installed: libxml2-python-2.9.1-5.el7_0.1.x86_64 Mar 28 13:47:32 rack1 yum[4170]: Installed: libxslt-python-1.1.28-5.el7.x86_64 Mar 28 13:47:33 rack1 yum[4170]: Installed: python-ncclient-0.4.2-2.el7.noarch Mar 28 13:47:35 rack1 yum[4170]: Updated: wxGTK-2.8.12-13.el7.x86_64 Mar 28 13:47:39 rack1 yum[4170]: Updated: python-neutron-2014.2.2-1.el7.noarch Mar 28 13:47:40 rack1 yum[4170]: Updated: openstack-neutron-2014.2.2-1.el7.noarch Mar 28 13:47:44 rack1 yum[4170]: Updated: python-nova-2014.2.2-1.el7.noarch Mar 28 13:47:44 rack1 yum[4170]: Updated: openstack-nova-common-2014.2.2-1.el7.noarch Mar 28 13:47:46 rack1 yum[4170]: Updated: python-glance-2014.2.2-1.el7.noarch Mar 28 13:47:47 rack1 yum[4170]: Updated: openstack-packstack-puppet-2014.2-0.18.dev1462.gbb05296.el7.noarch Mar 28 13:47:48 rack1 yum[4170]: Updated: novnc-0.5.1-2.el7.noarch Mar 28 13:47:53 rack1 yum[4170]: Updated: openstack-puppet-modules-2014.2.11-1.el7.noarch Mar 28 13:47:57 rack1 yum[4170]: Updated: python-django-horizon-2014.2.2-1.el7.noarch Mar 28 13:48:00 rack1 yum[4170]: Updated: python-ceilometer-2014.2.2-1.el7.noarch Mar 28 13:48:00 rack1 yum[4170]: Updated: openstack-ceilometer-common-2014.2.2-1.el7.noarch Mar 28 13:48:00 rack1 yum[4170]: Updated: openstack-ceilometer-notification-2014.2.2-1.el7.noarch Mar 28 13:48:06 rack1 yum[4170]: Updated: python-cinder-2014.2.2-1.el7.noarch Mar 28 13:48:08 rack1 yum[4170]: Updated: python-keystone-2014.2.2-1.el7.noarch Mar 28 13:48:10 rack1 yum[4170]: Updated: openstack-keystone-2014.2.2-1.el7.noarch Mar 28 13:48:14 rack1 yum[4170]: Updated: openstack-cinder-2014.2.2-1.el7.noarch Mar 28 13:48:15 rack1 yum[4170]: Updated: openstack-ceilometer-collector-2014.2.2-1.el7.noarch Mar 28 13:48:15 rack1 yum[4170]: Updated: openstack-ceilometer-central-2014.2.2-1.el7.noarch Mar 28 13:48:16 rack1 yum[4170]: Updated: openstack-ceilometer-api-2014.2.2-1.el7.noarch Mar 28 13:48:16 rack1 yum[4170]: Updated: openstack-ceilometer-alarm-2014.2.2-1.el7.noarch Mar 28 13:48:16 rack1 yum[4170]: Updated: openstack-ceilometer-compute-2014.2.2-1.el7.noarch Mar 28 13:48:24 rack1 yum[4170]: Updated: openstack-dashboard-2014.2.2-1.el7.noarch Mar 28 13:48:33 rack1 yum[4170]: Updated: openstack-packstack-2014.2-0.18.dev1462.gbb05296.el7.noarch Mar 28 13:48:34 rack1 yum[4170]: Updated: openstack-nova-novncproxy-2014.2.2-1.el7.noarch Mar 28 13:48:34 rack1 yum[4170]: Updated: openstack-glance-2014.2.2-1.el7.noarch Mar 28 13:48:35 rack1 yum[4170]: Updated: openstack-nova-compute-2014.2.2-1.el7.noarch Mar 28 13:48:36 rack1 yum[4170]: Updated: openstack-nova-api-2014.2.2-1.el7.noarch Mar 28 13:48:37 rack1 yum[4170]: Updated: openstack-nova-scheduler-2014.2.2-1.el7.noarch Mar 28 13:48:38 rack1 yum[4170]: Updated: openstack-nova-cert-2014.2.2-1.el7.noarch Mar 28 13:48:38 rack1 yum[4170]: Updated: openstack-nova-conductor-2014.2.2-1.el7.noarch Mar 28 13:48:39 rack1 yum[4170]: Updated: openstack-nova-console-2014.2.2-1.el7.noarch Mar 28 13:48:40 rack1 yum[4170]: Updated: openstack-neutron-openvswitch-2014.2.2-1.el7.noarch Mar 28 13:48:40 rack1 yum[4170]: Updated: openstack-neutron-ml2-2014.2.2-1.el7.noarch Mar 28 13:48:43 rack1 yum[4170]: Updated: wxGTK-gl-2.8.12-13.el7.x86_64 Mar 28 13:48:44 rack1 yum[4170]: Updated: facter-2.4.1-1.el7.x86_64 Mar 28 13:48:45 rack1 yum[4170]: Updated: figlet-2.2.5-7.el7.x86_64
Created attachment 1007904 [details] cinder/volume.log cinder/volume.log without raw html
After changing line 544 to LOG.errr("%s", unicode(e)), per @larks recommendation and restarting service, we get further. This may or may not be a proper fix, but it seems reasonable. The next challenge is this: 2015-03-30 15:29:28.064 4382 WARNING cinder.volume.manager [-] Unable to update volume replication status, LVMISCSIDriver -2.0.0 (config name lvm) driver is uninitialized. the full error trace is: http://fpaste.org/204967/77473851/ Ideas on how to proceed?
Would you mind opening a new bug on the new issue? Otherwise things get confused when we fix the original problem and close the bug and forget about the second (...and third, etc) problems. When opening the new bug, please don't link to pastebins per my comment in #3. Thanks!
Well, it looks like cinder and/or our combination of commands to try and get one of the volumes re-attached caused it to destroy the volume we wanted to attach, somehow. Somewhat upsetting, as we had some data there that will take a little bit to re-create and update from backups. So for now, this will be the bug.
This bug is against a version which has reached End of Life. Please reopen if it is still relevant with a latest version: http://releases.openstack.org/