I don't have a Ceph cluster at the moment.  Can you try testing
simpler qemu command lines to see what works, eg:

/usr/libexec/qemu-kvm \
    -drive snapshot=on,file=rbd:ssd-pool1/ceph-rbd-win08.2016110310320000.root:auth_supported=none,format=raw,id=hd0,if=ide

Ignore that, I didn't spot that there were two runs.

The libvirt XML is wrong as was established on the other bug,
so you're going to have to fix that first.  Add:
  <driver name='qemu' type='raw'/>
into the ceph disk.

Then use the -d option, and see what it says.

hi, (In reply to Richard W.M. Jones from comment #3)
> I don't have a Ceph cluster at the moment.  Can you try testing
> simpler qemu command lines to see what works, eg:
> 
> /usr/libexec/qemu-kvm \
>     -drive
> snapshot=on,file=rbd:ssd-pool1/ceph-rbd-win08.2016110310320000.root:
> auth_supported=none,format=raw,id=hd0,if=ide

[root@cnode1:/root]
# /usr/libexec/qemu-kvm -drive snapshot=on,file=rbd:ssd-pool1/ceph-rbd-win08.2016110310320000.root:auth_supported=none,format=raw,id=hd0,if=ide
qemu-kvm: -drive snapshot=on,file=rbd:ssd-pool1/ceph-rbd-win08.2016110310320000.root:auth_supported=none,format=raw,id=hd0,if=ide: error connecting: Operation not supported

(In reply to Richard W.M. Jones from comment #4)
> Ignore that, I didn't spot that there were two runs.
> 
> The libvirt XML is wrong as was established on the other bug,
> so you're going to have to fix that first.  Add:
>   <driver name='qemu' type='raw'/>
> into the ceph disk.
> 
> Then use the -d option, and see what it says.


i have add the type of the drive, the same err as using -a /disk.img option above

Hi,

is there any result？

Please try some different command lines to find out what exactly
doesn't work, see comment 4.

I mean, see comment 5, not comment 4.

can you give me a example？

Please modify the libvirt XML by adding:
  <driver name='qemu' type='raw'/>
into the Ceph disk in the libvirt XML.
Then run the test again using the '-d' option and see what it says.

my ceph disk used the type of raw，


    <disk type='network' device='disk'>
      <driver name='qemu' type='raw'/>
      <auth username='libvirt'>
        <secret type='ceph' uuid='d3af8319-14cd-49ca-a4d6-909ff4ce147f'/>
      </auth>
      <source protocol='rbd' name='ssd-pool1/ceph-rbd-win08.2016110310320000.root'>
        <host name='172.1.1.10' port='6789'/>
        <host name='172.1.1.11' port='6789'/>
        <host name='172.1.1.12' port='6789'/>
      </source>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw'/>
      <auth username='libvirt'>
        <secret type='ceph' uuid='d3af8319-14cd-49ca-a4d6-909ff4ce147f'/>
      </auth>
      <source protocol='rbd' name='ssd-pool1/ceph-rbd-win08.2016110310320000.data'>
        <host name='172.1.1.10' port='6789'/>
        <host name='172.1.1.11' port='6789'/>
        <host name='172.1.1.12' port='6789'/>
      </source>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>


still the same problem：

qemu-kvm: -drive file=rbd:ssd-pool1/ceph-rbd-win08.2016110310320000.root:mon_host=172.1.1.10\:6789\;172.1.1.11\:6789\;172.1.1.12\:6789:id=libvirt:auth_supported=cephx\;none,cache=writeback,format=raw,id=hd0,if=none: error connecting: Operation not supported
libguestfs: error: appliance closed the connection unexpectedly, see earlier error messages
libguestfs: child_cleanup: 0x7f6f11d0e940: child process died
libguestfs: sending SIGTERM to process 30928
libguestfs: error: /usr/libexec/qemu-kvm exited with error status 1, see debug messages above
libguestfs: error: guestfs_launch failed, see earlier error messages
libguestfs: trace: launch = -1 (error)
libguestfs: trace: close
libguestfs: closing guestfs handle 0x7f6f11d0e940 (state 0)
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /tmp/libguestfsBwHujS

any idea?

No, please follow the steps in comment 4, else there's nothing we
can do about this bug.

i have follow the steps in comment 4 already, the disk type has been changed to raw.

the result pls see the attachments

Created attachment 1220776 [details]
the output of guestfish inject

Created attachment 1220777 [details]
the xml of domain

The error is now completely different, and comes from libvirt.  Seems
to be something to do with the <secret> clause in the original guest
XML not matching any secret known by libvirt.

Original error from libvirt: XML error: missing auth secret uuid or usage attribute [code=27 int1=-1]

Thanks for taking the time to enter a bug report with us. We use reports like yours to keep improving the quality of our products and releases. That said, we're not able to guarantee the timeliness or suitability of a resolution for issues entered here because this is not a mechanism for requesting support.
                                                                                
If this issue is critical or in any way time sensitive, please raise a ticket through your regular Red Hat support channels to make certain it receives the proper attention and prioritization that will result in a timely resolution.
                                                                                
For information on how to contact the Red Hat production support team, please visit: https://www.redhat.com/support/process/production/#howto

(In reply to Richard W.M. Jones from comment #18)
> The error is now completely different, and comes from libvirt.  Seems
> to be something to do with the <secret> clause in the original guest
> XML not matching any secret known by libvirt.
> 
> Original error from libvirt: XML error: missing auth secret uuid or usage
> attribute [code=27 int1=-1]


but the domain can start and running well,.in the os,the rbd disk can work well.

it is proved that the secret must be right with libvirt and xml.

the libvirt secret uuid and xml are match

[root@cnode1:/root]
# virsh secret-list
 UUID                                  Usage
--------------------------------------------------------------------------------
 d3af8319-14cd-49ca-a4d6-909ff4ce147f  ceph client.libvirt secret


[root@cnode1:/root]
# virsh dumpxml ceph-rbd-win08|grep d3af
        <secret type='ceph' uuid='d3af8319-14cd-49ca-a4d6-909ff4ce147f'/>
        <secret type='ceph' uuid='d3af8319-14cd-49ca-a4d6-909ff4ce147f'/>

[root@cnode1:/root]
#

OK, it looks clearer to me now: libguestfs does not read the authentication/secret parts of disks at all (only the username), so opening them later on will fail (since there are no credentials provided).

Just sent a couple of patches (one cleanup, and the actual implementation) that should change this:
https://www.redhat.com/archives/libguestfs/2016-November/msg00080.html
https://www.redhat.com/archives/libguestfs/2016-November/msg00081.html

Thank you!

These two patches suitable for what version of libguestfs?

i use libguestfs-1.32.10

i tried to patch, but patch failed.


2 out of 11 hunks FAILED -- saving rejects to file src/libvirt-domain.c.rej




# cat src/libvirt-domain.c.rej
--- src/libvirt-domain.c
+++ src/libvirt-domain.c
@@ -42,7 +44,7 @@
 #if defined(HAVE_LIBVIRT)

 static xmlDocPtr get_domain_xml (guestfs_h *g, virDomainPtr dom);
-static ssize_t for_each_disk (guestfs_h *g, virConnectPtr conn, xmlDocPtr doc, int (*f) (guestfs_h *g, const char *filename, const char *format, int readonly, const char *protocol, char *const *server, const char *username, void *data), void *data);
+static ssize_t for_each_disk (guestfs_h *g, virConnectPtr conn, xmlDocPtr doc, int (*f) (guestfs_h *g, const char *filename, const char *format, int readonly, const char *protocol, char *const *server, const char *username, const char *secret, void *data), void *data);
 static int libvirt_selinux_label (guestfs_h *g, xmlDocPtr doc, char **label_rtn, char **imagelabel_rtn);
 static char *filename_from_pool (guestfs_h *g, virConnectPtr conn, const char *pool_nane, const char *volume_name);
 static bool xPathObjectIsEmpty (xmlXPathObjectPtr obj);
@@ -580,8 +591,111 @@
         xpusername = xmlXPathEvalExpression (BAD_CAST "./auth/@username",
                                              xpathCtx);
         if (!xPathObjectIsEmpty (xpusername)) {
+          CLEANUP_XMLXPATHFREEOBJECT xmlXPathObjectPtr xpsecrettype = NULL;
+          CLEANUP_XMLXPATHFREEOBJECT xmlXPathObjectPtr xpsecretuuid = NULL;
+          CLEANUP_XMLXPATHFREEOBJECT xmlXPathObjectPtr xpsecretusage = NULL;
+          CLEANUP_FREE char *typestr = NULL;
+          unsigned char *value = NULL;
+          size_t value_size = 0;
+
           username = xPathObjectGetString (doc, xpusername);
           debug (g, "disk[%zu]: username: %s", i, username);
+
+          /* <secret type="...">.  Mandatory given <auth> is specified. */
+          xpsecrettype = xmlXPathEvalExpression (BAD_CAST "./auth/secret/@type",
+                                                 xpathCtx);
+          if (xPathObjectIsEmpty (xpsecrettype))
+            continue;
+          typestr = xPathObjectGetString (doc, xpsecrettype);
+
+          /* <secret uuid="..."> and <secret usage="...">.
+           * At least one of them is required.
+           */
+          xpsecretuuid = xmlXPathEvalExpression (BAD_CAST "./auth/secret/@uuid",
+                                                 xpathCtx);
+          xpsecretusage = xmlXPathEvalExpression (BAD_CAST "./auth/secret/@usage",
+                                                  xpathCtx);
+          if (!xPathObjectIsEmpty (xpsecretuuid)) {
+            CLEANUP_FREE char *uuidstr = NULL;
+            virSecretPtr sec;
+
+            uuidstr = xPathObjectGetString (doc, xpsecretuuid);
+            debug (g, "disk[%zu]: secret type: %s; UUID: %s",
+                   i, typestr, uuidstr);
+            sec = virSecretLookupByUUIDString (conn, uuidstr);
+            if (sec == NULL) {
+              err = virGetLastError ();
+              error (g, _("no secret with UUID '%s': %s"),
+                     uuidstr, err ? err->message : "(none)");
+              continue;
+            }
+
+            value = virSecretGetValue (sec, &value_size, 0);
+            if (value == NULL) {
+              err = virGetLastError ();
+              error (g, _("cannot get the value of the secret with UUID '%s': %s"),
+                     uuidstr, err->message);
+              virSecretFree (sec);
+              continue;
+            }
+
+            virSecretFree (sec);
+          } else if (!xPathObjectIsEmpty (xpsecretusage)) {
+            virSecretUsageType usageType;
+            CLEANUP_FREE char *usagestr = NULL;
+            virSecretPtr sec;
+
+            usagestr = xPathObjectGetString (doc, xpsecretusage);
+            debug (g, "disk[%zu]: secret type: %s; usage: %s",
+                   i, typestr, usagestr);
+            if (STREQ (usagestr, "none"))
+              usageType = VIR_SECRET_USAGE_TYPE_NONE;
+            else if (STREQ (usagestr, "volume"))
+              usageType = VIR_SECRET_USAGE_TYPE_VOLUME;
+            else if (STREQ (usagestr, "ceph"))
+              usageType = VIR_SECRET_USAGE_TYPE_CEPH;
+            else if (STREQ (usagestr, "iscsi"))
+              usageType = VIR_SECRET_USAGE_TYPE_ISCSI;
+            else
+              continue;
+            sec = virSecretLookupByUsage (conn, usageType, usagestr);
+            if (sec == NULL) {
+              err = virGetLastError ();
+              error (g, _("no secret for usage '%s': %s"),
+                     usagestr, err->message);
+              continue;
+            }
+
+            value = virSecretGetValue (sec, &value_size, 0);
+            if (value == NULL) {
+              err = virGetLastError ();
+              error (g, _("cannot get the value of the secret with usage '%s': %s"),
+                     usagestr, err->message);
+              virSecretFree (sec);
+              continue;
+            }
+
+            virSecretFree (sec);
+          } else {
+            continue;
+          }
+
+          assert (value != NULL);
+          assert (value_size > 0);
+
+          if (STREQ (typestr, "ceph")) {
+            const size_t res = base64_encode_alloc ((const char *) value,
+                                                    value_size, &secret);
+            free (value);
+            if (res == 0 || secret == NULL) {
+              error (g, "internal error: cannot encode the rbd secret as base64");
+              return -1;
+            }
+          } else {
+            secret = (char *) value;
+          }
+
+          assert (secret != NULL);
         }

         xphost = xmlXPathEvalExpression (BAD_CAST "./source/host",

(In reply to 395783748 from comment #22)
> Thank you!
> 
> These two patches suitable for what version of libguestfs?

They apply on current git/master, which currently is 1.35.x. Also, there were other commits on that file, so it widens the difference to 1.32.x.
I'll provide you a single diff of the proposed fix plus all the needed patches for it, that applies on top of 1.32.x.

Created attachment 1222686 [details]
[PATCH] proposed fix + needed patches

This patch is for libguestfs 1.32.x, and includes:
- 4c3968f262e8a45f65f8980d6af39144bd52f0ea (small parts, mostly the virConnect stuff)
- the two patches linked in comment 21

Fixed with
https://github.com/libguestfs/libguestfs/commit/bef838202b533aa008e62af3f78e0c4654b7c5e9 (cleanup)
https://github.com/libguestfs/libguestfs/commit/a94d5513456d7255d6e562953ac163f2d7a816fb
which are in libguestfs >= 1.35.15.

(In reply to Pino Toscano from comment #25)
> Fixed with
> https://github.com/libguestfs/libguestfs/commit/
> bef838202b533aa008e62af3f78e0c4654b7c5e9 (cleanup)
> https://github.com/libguestfs/libguestfs/commit/
> a94d5513456d7255d6e562953ac163f2d7a816fb
> which are in libguestfs >= 1.35.15.

... and also a followup/fix:
https://github.com/libguestfs/libguestfs/commit/7bd6a73f0092cf1e23f9b0584a3212df5309367c
which is in libguestfs >= 1.35.18.

Can reproduce it with:
libguestfs-1.32.7-3.el7.x86_64

Steps:
1. Prepare ceph server according to :
Refer to: https://drive.google.com/open?id=1ryfe-6D968kEiy2YzloMnLXC0tw1cfmeGeIKBndpbUg

In this bug, we use 10.66.10.242 as the ceph mon.
Create a pool: libvirt-pool

2. on mon node:
# cat /etc/ceph/ceph.conf
... ...
auth cluster required = cephx
auth service required = cephx
auth client required = cephx

# ceph auth get-or-create client.libvirt mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=libvirt-pool'
[[client.libvirt]
    key = AQDnPkpYgg4hIRAAM3z67RZ1spc28zAVi0XC6w==

# ceph auth list
client.libvirt
    key: AQDnPkpYgg4hIRAAM3z67RZ1spc28zAVi0XC6w==
    caps: [mon] allow r
    caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=libvirt-pool

3. On client:
# vim secret.xml
<secret ephemeral='no' private='no'>
    <description>CEPH passphrase example</description>
        <usage type='ceph'>
          <name>client.libvirt secret</name>
        </usage>
</secret>

# virsh secret-define secret.xml
Secret 27be818b-9248-40e3-b0a9-706e3ae72925 created

#  virsh secret-list
UUID                                  Usage
--------------------------------------------------------------------------------
 27be818b-9248-40e3-b0a9-706e3ae72925  ceph client.libvirt secret

# virsh secret-set-value --secret 27be818b-9248-40e3-b0a9-706e3ae72925 --base64 AQDnPkpYgg4hIRAAM3z67RZ1spc28zAVi0XC6w==

4. On client, Prepare a guest image:rbd-secret.img
# qemu-img create -f raw rbd:libvirt-pool/rbd-secret.img:id=libvirt:key=AQDnPkpYgg4hIRAAM3z67RZ1spc28zAVi0XC6w==:mon_host=10.66.10.242 8G

Create and start a rhel7.2 guest with the following ceph-secret.xml:
... ...
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<auth username='libvirt'>
<secret type='ceph' usage='client.libvirt secret'/>
</auth>
<source protocol='rbd' name='libvirt-pool/rbd-secret.img'>
<host name='10.66.10.242' port='6789'/>
</source>
<backingStore/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
</disk>
... ...

# vish create  ceph-secret.xml
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 23    ceph-secret                    running

5. On client, Use guestfish to access the guest image:
# guestfish -d ceph-secret --ro -i
libguestfs: error: qemu-img: /tmp/libguestfsBs0eXd/overlay1: qemu-img exited with error status 1.
To see full error messages you may need to enable debugging.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.


So, this bug can be reproduced.

Verified with package:
libguestfs-1.36.3-1.el7.x86_64

Steps:
1. Prepare ceph server according to :
Refer to: https://drive.google.com/open?id=1ryfe-6D968kEiy2YzloMnLXC0tw1cfmeGeIKBndpbUg

Here we use 10.66.144.75 as the ceph mon.


2. on mon node:
#ceph osd pool create libvirt-pool 128 128

# ceph auth get-or-create client.libvirt mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=libvirt-pool'
[[client.libvirt]
    key = AQCEcuhY4eicORAAR65g5TTjL9086ltA1Lbmfg==

# ceph auth list
client.libvirt
    key: AQCEcuhY4eicORAAR65g5TTjL9086ltA1Lbmfg==
    caps: [mon] allow r
    caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=libvirt-pool

3. On client:
# vim secret.xml
<secret ephemeral='no' private='no'>
    <description>CEPH passphrase example</description>
        <usage type='ceph'>
          <name>client.libvirt secret</name>
        </usage>
</secret>

# virsh secret-define secret.xml
Secret 27be818b-9248-40e3-b0a9-706e3ae72925 created

#  virsh secret-list
UUID                                  Usage
--------------------------------------------------------------------------------
 b710e6bf-de07-4cef-bef9-cad0ee06ee2e  ceph client.libvirt secret

# virsh secret-set-value --secret b710e6bf-de07-4cef-bef9-cad0ee06ee2e --base64 AQDnPkpYgg4hIRAAM3z67RZ1spc28zAVi0XC6w==

4. On client, Prepare a guest image:rbd-secret.img
# qemu-img create -f raw rbd:libvirt-pool/rbd-secret.img:id=libvirt:key=AQCEcuhY4eicORAAR65g5TTjL9086ltA1Lbmfg==:mon_host=10.66.144.75 8G

Create a rhel7.3 vm on rbd-secret.img with the following ceph-secret.xml:
... ...
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<auth username='libvirt'>
<secret type='ceph' usage='client.libvirt secret'/>
</auth>
<source protocol='rbd' name='libvirt-pool/rbd-secret.img'>
<host name='10.66.144.75' port='6789'/>
</source>
<backingStore/>
<target dev='hda' bus='ide'/>
</disk>
... ...

# vish create  ceph-secret.xml
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 23    ceph-secret                    running

5. On client, Use guestfish to access the guest image:
# guestfish -d ceph-secret --ro
><fs> run
><fs> list-filesystems
/dev/sda1: xfs
/dev/rhel/root: xfs
/dev/rhel/swap: swap

From the results above, the guest image can be inspected via guestfish.
So veriried this bug.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2023