Bug 1206864

Summary:

cannot attach local cinder volume

Product:

[Community] RDO

Reporter:

cpg <cpg>

Component:

openstack-cinder

Assignee:

Eric Harney <eharney>

Status:

CLOSED EOL

QA Contact:

nlevinki <nlevinki>

Severity:

unspecified

Docs Contact:

Priority:

unspecified

Version:

Juno

CC:

amoralej, eharney, lars, srevivo

Target Milestone:

---

Target Release:

Kilo

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2016-05-19 08:13:01 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

1206867

Attachments:

Description	Flags
yum.log	none
cinder volume.log	none
cinder/volume.log	none

Description cpg 2015-03-29 02:47:16 UTC

Description of problem:

After recent updates, we cannot re-attach volumes.

Version-Release number of selected component (if applicable):

We just run RDO (Juno) with Centos 7, and we usually wait a week or two to apply updates.

How reproducible:

Hard to tell. We're stuck on this situation and we cannot go back to try, as yum downgrade failed.

Steps to Reproduce:
1. Run RDO. Stop instances.
2. Apply these recent updates today http://fpaste.org/204340/27597219/
3. Reboot
4. Try to bring things up, and the attach of a local volume fails

Actual results:

this is the error in cinder/volume.log: http://fpaste.org/204339/14275970/

Expected results:

Be able to run our instances again!

Additional info:

I tried to diagnose if the issue was with logging (after seeing the Oslo people report that logging has to change across libraries to use unicode), by comenting out this line 522 of /usr/lib/python2.7/site-packages/cinder/brick/iscsi/iscsi.py:

          LOG.info(_('Creating iscsi_target for volume: %s') % vol_id)

other than that, nothing else is changed.

Comment 1 Lars Kellogg-Stedman 2015-03-29 02:49:18 UTC

Created attachment 1007901 [details]
yum.log

Comment 2 Lars Kellogg-Stedman 2015-03-29 02:49:44 UTC

Created attachment 1007902 [details]
cinder volume.log

Comment 3 Lars Kellogg-Stedman 2015-03-29 02:50:56 UTC

Thanks for the report!

We try to discourage people from referring to pastebins in bug reports, because when the paste expires the bug becomes less useful.  I've attached your two pastes to this bug, so you're all set for now.

Comment 4 cpg 2015-03-29 05:25:41 UTC

yum.log, without html

Mar 28 13:47:31 rack1 yum[4170]: Updated: wxBase-2.8.12-13.el7.x86_64
Mar 28 13:47:32 rack1 yum[4170]: Installed: libxml2-python-2.9.1-5.el7_0.1.x86_64
Mar 28 13:47:32 rack1 yum[4170]: Installed: libxslt-python-1.1.28-5.el7.x86_64
Mar 28 13:47:33 rack1 yum[4170]: Installed: python-ncclient-0.4.2-2.el7.noarch
Mar 28 13:47:35 rack1 yum[4170]: Updated: wxGTK-2.8.12-13.el7.x86_64
Mar 28 13:47:39 rack1 yum[4170]: Updated: python-neutron-2014.2.2-1.el7.noarch
Mar 28 13:47:40 rack1 yum[4170]: Updated: openstack-neutron-2014.2.2-1.el7.noarch
Mar 28 13:47:44 rack1 yum[4170]: Updated: python-nova-2014.2.2-1.el7.noarch
Mar 28 13:47:44 rack1 yum[4170]: Updated: openstack-nova-common-2014.2.2-1.el7.noarch
Mar 28 13:47:46 rack1 yum[4170]: Updated: python-glance-2014.2.2-1.el7.noarch
Mar 28 13:47:47 rack1 yum[4170]: Updated: openstack-packstack-puppet-2014.2-0.18.dev1462.gbb05296.el7.noarch
Mar 28 13:47:48 rack1 yum[4170]: Updated: novnc-0.5.1-2.el7.noarch
Mar 28 13:47:53 rack1 yum[4170]: Updated: openstack-puppet-modules-2014.2.11-1.el7.noarch
Mar 28 13:47:57 rack1 yum[4170]: Updated: python-django-horizon-2014.2.2-1.el7.noarch
Mar 28 13:48:00 rack1 yum[4170]: Updated: python-ceilometer-2014.2.2-1.el7.noarch
Mar 28 13:48:00 rack1 yum[4170]: Updated: openstack-ceilometer-common-2014.2.2-1.el7.noarch
Mar 28 13:48:00 rack1 yum[4170]: Updated: openstack-ceilometer-notification-2014.2.2-1.el7.noarch
Mar 28 13:48:06 rack1 yum[4170]: Updated: python-cinder-2014.2.2-1.el7.noarch
Mar 28 13:48:08 rack1 yum[4170]: Updated: python-keystone-2014.2.2-1.el7.noarch
Mar 28 13:48:10 rack1 yum[4170]: Updated: openstack-keystone-2014.2.2-1.el7.noarch
Mar 28 13:48:14 rack1 yum[4170]: Updated: openstack-cinder-2014.2.2-1.el7.noarch
Mar 28 13:48:15 rack1 yum[4170]: Updated: openstack-ceilometer-collector-2014.2.2-1.el7.noarch
Mar 28 13:48:15 rack1 yum[4170]: Updated: openstack-ceilometer-central-2014.2.2-1.el7.noarch
Mar 28 13:48:16 rack1 yum[4170]: Updated: openstack-ceilometer-api-2014.2.2-1.el7.noarch
Mar 28 13:48:16 rack1 yum[4170]: Updated: openstack-ceilometer-alarm-2014.2.2-1.el7.noarch
Mar 28 13:48:16 rack1 yum[4170]: Updated: openstack-ceilometer-compute-2014.2.2-1.el7.noarch
Mar 28 13:48:24 rack1 yum[4170]: Updated: openstack-dashboard-2014.2.2-1.el7.noarch
Mar 28 13:48:33 rack1 yum[4170]: Updated: openstack-packstack-2014.2-0.18.dev1462.gbb05296.el7.noarch
Mar 28 13:48:34 rack1 yum[4170]: Updated: openstack-nova-novncproxy-2014.2.2-1.el7.noarch
Mar 28 13:48:34 rack1 yum[4170]: Updated: openstack-glance-2014.2.2-1.el7.noarch
Mar 28 13:48:35 rack1 yum[4170]: Updated: openstack-nova-compute-2014.2.2-1.el7.noarch
Mar 28 13:48:36 rack1 yum[4170]: Updated: openstack-nova-api-2014.2.2-1.el7.noarch
Mar 28 13:48:37 rack1 yum[4170]: Updated: openstack-nova-scheduler-2014.2.2-1.el7.noarch
Mar 28 13:48:38 rack1 yum[4170]: Updated: openstack-nova-cert-2014.2.2-1.el7.noarch
Mar 28 13:48:38 rack1 yum[4170]: Updated: openstack-nova-conductor-2014.2.2-1.el7.noarch
Mar 28 13:48:39 rack1 yum[4170]: Updated: openstack-nova-console-2014.2.2-1.el7.noarch
Mar 28 13:48:40 rack1 yum[4170]: Updated: openstack-neutron-openvswitch-2014.2.2-1.el7.noarch
Mar 28 13:48:40 rack1 yum[4170]: Updated: openstack-neutron-ml2-2014.2.2-1.el7.noarch
Mar 28 13:48:43 rack1 yum[4170]: Updated: wxGTK-gl-2.8.12-13.el7.x86_64
Mar 28 13:48:44 rack1 yum[4170]: Updated: facter-2.4.1-1.el7.x86_64
Mar 28 13:48:45 rack1 yum[4170]: Updated: figlet-2.2.5-7.el7.x86_64

Comment 5 cpg 2015-03-29 05:27:47 UTC

Created attachment 1007904 [details]
cinder/volume.log

cinder/volume.log without raw html

Comment 6 cpg 2015-03-30 20:41:42 UTC

After changing line 544 to LOG.errr("%s", unicode(e)), per @larks recommendation and restarting service, we get further. This may or may not be a proper fix, but it seems reasonable.

The next challenge is this:

2015-03-30 15:29:28.064 4382 WARNING cinder.volume.manager [-] Unable to update volume replication status, LVMISCSIDriver -2.0.0 (config name lvm) driver is uninitialized.

the full error trace is: http://fpaste.org/204967/77473851/

Ideas on how to proceed?

Comment 7 Lars Kellogg-Stedman 2015-03-30 20:54:39 UTC

Would you mind opening a new bug on the new issue?  Otherwise things get confused when we fix the original problem and close the bug and forget about the second (...and third, etc) problems.

When opening the new bug, please don't link to pastebins per my comment in #3.  Thanks!

Comment 8 cpg 2015-03-31 06:30:06 UTC

Well, it looks like cinder and/or our combination of commands to try and get one of the volumes re-attached caused it to destroy the volume we wanted to attach, somehow. Somewhat upsetting, as we had some data there that will take a little bit to re-create and update from backups.

So for now, this will be the bug.

Comment 9 Alfredo Moralejo 2016-05-19 08:13:01 UTC

This bug is against a version which has reached End of Life. Please reopen if it is still relevant with a latest version: http://releases.openstack.org/