1672620 – Failed to attache NEW rbd device to guest

Bug 1672620 - Failed to attache NEW rbd device to guest

Summary: Failed to attache NEW rbd device to guest

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Fedora
Classification:	Fedora
Component:	libvirt
Sub Component:
Version:	29
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Assignee:	Libvirt Maintainers
QA Contact:	Fedora Extras Quality Assurance
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-02-05 12:52 UTC by Attila Fazekas
Modified:	2019-07-09 02:24 UTC (History)
CC List:	10 users (show)
Fixed In Version:	libvirt-4.7.0-5.fc29
Clone Of:
Environment:
Last Closed:	2019-07-09 02:24:23 UTC
Type:	Bug
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description Attila Fazekas 2019-02-05 12:52:10 UTC

Description of problem:
I am using the development version of nova,
The instances are configured to use ceph/rbd for instance store (root disk) and it is working,
but attaching a 2th volume to a live guest is not working!

The cinder/nova api shows the attach happened, nothing indicate error in their logs,

The dumped domain xml did showed the 2th drive, the guest does not see it either.

libvirt log:
qemuDomainGetSecretAESAlias:788 : invalid argument: encrypted secret alias requires valid source alias
2019-02-05 11:34:55.598+0000: 799: debug : virThreadJobClear:121 : Thread 799 (virNetServerHandleJob) finished job remoteDispatchDomainAttachDeviceFlags with ret=-1

Version-Release number of selected component (if applicable):
libvirt-daemon-4.7.0-1.fc29.x86_64


How reproducible:
always

Steps to Reproduce:
run tempest basic minimum scenario (stestr run minimum), it will fail on detecting the new volume on the guest (ssh/list disk).

libvirt debug log:

2019-02-05 11:34:55.578+0000: 799: debug : virDomainAttachDeviceFlags:8147 : dom=0x7ff6700446e0, (VM: name=instance-00000044, uuid=f098bec3-2c65-4f2d-8a89-0c74593b7539), xml=<disk type="network" device="disk">
  <driver name="qemu" type="raw" cache="writeback" discard="unmap"/>
  <source protocol="rbd" name="volumes/volume-8fad12bc-d119-4bd4-85a8-5d4f2ece6cdd">
    <host name="172.16.1.2" port="6789"/>
  </source>
  <auth username="cinder">
    <secret type="ceph" uuid="457eb676-33da-42ec-9a8c-9293d545c337"/>
  </auth>
  <target bus="virtio" dev="vdb"/>
  <serial>8fad12bc-d119-4bd4-85a8-5d4f2ece6cdd</serial>
</disk>
, flags=0x3


2019-02-05 11:34:55.598+0000: 799: debug : virConnectOpenInternal:1019 : Matched URI scheme 'secret'
2019-02-05 11:34:55.598+0000: 799: info : virObjectRef:382 : OBJECT_REF: obj=0x561db6ce35d0
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x561db6ce35d0
2019-02-05 11:34:55.598+0000: 799: debug : virConnectOpenInternal:1052 : driver 11 secret returned SUCCESS
2019-02-05 11:34:55.598+0000: 799: error : qemuDomainGetSecretAESAlias:788 : invalid argument: encrypted secret alias requires valid source alias
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7ff67002d990
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:346 : OBJECT_DISPOSE: obj=0x7ff67002d990
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7ff634116730
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7ff6700123b0
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:346 : OBJECT_DISPOSE: obj=0x7ff6700123b0
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7ff67001a360
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:346 : OBJECT_DISPOSE: obj=0x7ff67001a360
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7ff670044720
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:346 : OBJECT_DISPOSE: obj=0x7ff670044720
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7ff670094ef0
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:346 : OBJECT_DISPOSE: obj=0x7ff670094ef0
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7ff670094310
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:346 : OBJECT_DISPOSE: obj=0x7ff670094310
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7ff6700102f0
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:346 : OBJECT_DISPOSE: obj=0x7ff6700102f0
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7ff6700101d0
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:346 : OBJECT_DISPOSE: obj=0x7ff6700101d0
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7ff670021550
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:346 : OBJECT_DISPOSE: obj=0x7ff670021550
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7ff670095790
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:346 : OBJECT_DISPOSE: obj=0x7ff670095790
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7ff634116730
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7ff6740560b0
2019-02-05 11:34:55.598+0000: 799: debug : qemuDomainObjEndJob:7054 : Stopping job: modify (async=none vm=0x7ff67c015480 name=instance-00000044)
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7ff67c015480
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7ff6700446e0
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:346 : OBJECT_DISPOSE: obj=0x7ff6700446e0
2019-02-05 11:34:55.598+0000: 799: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7ff67403b1a0
2019-02-05 11:34:55.598+0000: 799: debug : virThreadJobClear:121 : Thread 799 (virNetServerHandleJob) finished job remoteDispatchDomainAttachDeviceFlags with ret=-1


Additional info:
Upgrading libvirt to libvirt-4.10.0-2.fc30.x86_64 solved the issue,
and it was working thing with f28 ..

Comment 1 Peter Krempa 2019-02-05 13:20:51 UTC

This was broken by commit:

commit 192fdaa614e3800255048a8a70c1292ccf18397a
Author: Peter Krempa <pkrempa>
Date:   Mon Aug 13 15:17:36 2018 +0200

    qemu: hotplug: Prepare disk source in qemuDomainAttachDeviceDiskLive
    
    Move the preparation steps from qemuDomainAttachDiskGeneric up into
    qemuDomainAttachDeviceDiskLive so that also media changing can use the
    prepared file.

v4.6.0-220-g192fdaa614, thus released in v4.7.0

which moved the secret preparation code prior to the disk alias being assigned. The disk alias in this case is used as source for generating the secret alias name so the above method fails.

It was fixed upstream by:
commit 9ac196997839a29486029a02d8f519df54ae0186
Author: Peter Krempa <pkrempa>
Date:   Mon Sep 24 16:49:01 2018 +0200

    Revert "qemu: hotplug: Prepare disk source in qemuDomainAttachDeviceDiskLive"
    
    Preparing the storage source prior to assigning the alias will not work
    as the names of the certain objects depend on the alias for the legacy
    hotplug case as we generate the object names for the secrets based on
    the alias.
    
    This reverts commit 192fdaa614e3800255048a8a70c1292ccf18397a.

v4.8.0-79-g9ac1969978, thus released in v4.9.0.

Comment 2 Attila Fazekas 2019-02-05 13:33:07 UTC

I tested 2 version in the meantime,
v4.9.0 was working, v4.8.0 is broken too, as you said.

Comment 3 Fedora Update System 2019-06-20 22:38:52 UTC

FEDORA-2019-9210998aaa has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-9210998aaa

Comment 4 Fedora Update System 2019-06-22 02:46:06 UTC

libvirt-4.7.0-5.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-9210998aaa

Comment 5 Fedora Update System 2019-07-09 02:24:23 UTC

libvirt-4.7.0-5.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.