Bug 1222058 - hosted-engine fails to transfer the appliance image to the storage domain due to bad permissions
Summary: hosted-engine fails to transfer the appliance image to the storage domain due...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: oVirt
Classification: Retired
Component: vdsm
Version: 3.6
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 3.6.0
Assignee: Sandro Bonazzola
QA Contact: Elad
URL:
Whiteboard: integration
Depends On:
Blocks: 1159166 1215623 1234915
TreeView+ depends on / blocked
 
Reported: 2015-05-15 15:57 UTC by Simone Tiraboschi
Modified: 2015-11-04 13:43 UTC (History)
19 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, when a self-hosted engine is deployed over iSCSI using the oVirt Virtual Appliance, the installation fails because the logical volume was not automatically activated after the creation of the Manager virtual machine image. With this update, the logical volume is automatically activated and the deployment proceeds as normal.
Clone Of:
Environment:
Last Closed: 2015-11-04 13:43:57 UTC
oVirt Team: ---
Embargoed:


Attachments (Terms of Use)
vdsm logs (74.87 KB, application/x-gzip)
2015-05-15 16:13 UTC, Simone Tiraboschi
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 41964 0 master ABANDONED packaging: setup: prepareImage before uploading Never
oVirt gerrit 41965 0 master MERGED Add missing call to prepareImage Never
oVirt gerrit 44618 0 ovirt-hosted-engine-setup-1.2 MERGED Add missing call to prepareImage Never

Description Simone Tiraboschi 2015-05-15 15:57:51 UTC
Description of problem:
Try to deploy hosted-engine over iSCSI booting from oVirt appliance, it fails transferring the appliance image to the selected iSCSI device.

[ INFO  ] Creating VM Image
[ INFO  ] Extracting disk image from OVF archive (could take a few minutes depending on archive size)
[ INFO  ] Validating pre-allocated volume size
[ INFO  ] Uploading volume to data domain (could take a few minutes depending on archive size)
[ ERROR ] Failed to execute stage 'Misc configuration': Command '/bin/sudo' failed to execute

And from hosted-engine-setup logs:
2015-05-15 17:24:31 DEBUG otopi.plugins.ovirt_hosted_engine_setup.vm.boot_disk plugin.execute:940 execute-output: ('/bin/sudo', '-u', 'vdsm', '-g', 'kvm', '/bin/qemu-img', 'convert', '-O', 'raw', '/tmp/tmpHKCpAE', '/rhev/data-center/mnt/blockSD/1bfb9191-9acf-4637-851c-7e19da55445e/images/5fd6a8f0-35e0-4ef1-b9ce-5560054b8e47/0e78f967-2a11-4ce6-bfe1-d0409ece40db') stderr:
qemu-img: /rhev/data-center/mnt/blockSD/1bfb9191-9acf-4637-851c-7e19da55445e/images/5fd6a8f0-35e0-4ef1-b9ce-5560054b8e47/0e78f967-2a11-4ce6-bfe1-d0409ece40db: error while converting raw: Could not create file: Permission denied

and indeed also
[root@c71he36is1 ~]# sudo -u vdsm -g kvm /bin/qemu-img convert -O raw /mnt/ovirt.ova /rhev/data-center/mnt/blockSD/1bfb9191-9acf-4637-851c-7e19da55445e/images/5fd6a8f0-35e0-4ef1-b9ce-5560054b8e47/0e78f967-2a11-4ce6-bfe1-d0409ece40db
qemu-img: /rhev/data-center/mnt/blockSD/1bfb9191-9acf-4637-851c-7e19da55445e/images/5fd6a8f0-35e0-4ef1-b9ce-5560054b8e47/0e78f967-2a11-4ce6-bfe1-d0409ece40db: error while converting raw: Could not create file: Permission denied
fails

while the same command works if executed as root

On the permission side I have:
[root@c71he36is1 ~]# ls -l /rhev/data-center/mnt/blockSD/1bfb9191-9acf-4637-851c-7e19da55445e/images/5fd6a8f0-35e0-4ef1-b9ce-5560054b8e47/
total 0
lrwxrwxrwx. 1 vdsm kvm 78 May 15 17:23 0e78f967-2a11-4ce6-bfe1-d0409ece40db -> /dev/1bfb9191-9acf-4637-851c-7e19da55445e/0e78f967-2a11-4ce6-bfe1-d0409ece40db

[root@c71he36is1 ~]# ls -l /dev/1bfb9191-9acf-4637-851c-7e19da55445e/0e78f967-2a11-4ce6-bfe1-d0409ece40db
-rw-r--r--. 1 root root 993832960 May 15 17:47 /dev/1bfb9191-9acf-4637-851c-7e19da55445e/0e78f967-2a11-4ce6-bfe1-d0409ece40db

and if I change (not sure if it's really a good idea) the ownership of /dev/1bfb9191-9acf-4637-851c-7e19da55445e/0e78f967-2a11-4ce6-bfe1-d0409ece40db
to vdsm:kvm also

[root@c71he36is1 ~]# chown vdsm:kvm /dev/1bfb9191-9acf-4637-851c-7e19da55445e/0e78f967-2a11-4ce6-bfe1-d0409ece40db
[root@c71he36is1 ~]# sudo -u vdsm -g kvm /bin/qemu-img convert -O raw /mnt/ovirt.ova /rhev/data-center/mnt/blockSD/1bfb9191-9acf-4637-851c-7e19da55445e/images/5fd6a8f0-35e0-4ef1-b9ce-5560054b8e47/0e78f967-2a11-4ce6-bfe1-d0409ece40db
[root@c71he36is1 ~]# ls -l /dev/1bfb9191-9acf-4637-851c-7e19da55445e/0e78f967-2a11-4ce6-bfe1-d0409ece40db-rw-r--r--. 1 vdsm kvm 993832960 May 15 17:56 /dev/1bfb9191-9acf-4637-851c-7e19da55445e/0e78f967-2a11-4ce6-bfe1-d0409ece40db

works as expected



Version-Release number of selected component (if applicable):
vdsm.x86_64                       4.17.0-792.gitfa9c669.el7            @ovirt-master-snapshot

How reproducible:
100%

Steps to Reproduce:
1. try to deploy hosted engine over iSCSI using engine appliance
2.
3.

Actual results:
It fails:
[ INFO  ] Creating VM Image
[ INFO  ] Extracting disk image from OVF archive (could take a few minutes depending on archive size)
[ INFO  ] Validating pre-allocated volume size
[ INFO  ] Uploading volume to data domain (could take a few minutes depending on archive size)
[ ERROR ] Failed to execute stage 'Misc configuration': Command '/bin/sudo' failed to execute
[ INFO  ] Stage: Clean up


Expected results:
It succeeds

Additional info:
Seen on Centos 7.1, SELinux in permissive mode

Comment 1 Simone Tiraboschi 2015-05-15 16:13:19 UTC
Created attachment 1025935 [details]
vdsm logs

vdsm logs

Comment 2 Allon Mureinik 2015-05-17 08:33:34 UTC
Is the LV activated properly?

Comment 3 Simone Tiraboschi 2015-05-18 07:53:53 UTC
(In reply to Allon Mureinik from comment #2)
> Is the LV activated properly?

I don't think so:
I didn't found any LV on /dev/1bfb9191-9acf-4637-851c-7e19da55445e while I found something slightly different.

[root@c71he36is1 ~]# lvdisplay 
  --- Logical volume ---
  LV Path                /dev/ef720393-a513-4ce0-a5a7-43a3b2e04fc7/metadata
  LV Name                metadata
  VG Name                ef720393-a513-4ce0-a5a7-43a3b2e04fc7
  LV UUID                yfYv2V-xd8v-eEIb-FeKf-fMLx-ghaG-e5AFTF
  LV Write Access        read/write
  LV Creation host, time f20tre36i.localdomain, 2015-04-01 16:39:12 +0200
  LV Status              available
  # open                 0
  LV Size                512,00 MiB
  Current LE             4
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:7
   
  --- Logical volume ---
  LV Path                /dev/ef720393-a513-4ce0-a5a7-43a3b2e04fc7/outbox
  LV Name                outbox
  VG Name                ef720393-a513-4ce0-a5a7-43a3b2e04fc7
  LV UUID                Mmh7ZW-FHCF-gbXe-vJuW-5l0M-CSg9-9myCKg
  LV Write Access        read/write
  LV Creation host, time f20tre36i.localdomain, 2015-04-01 16:39:12 +0200
  LV Status              available
  # open                 0
  LV Size                128,00 MiB
  Current LE             1
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:8
   
  --- Logical volume ---
  LV Path                /dev/ef720393-a513-4ce0-a5a7-43a3b2e04fc7/leases
  LV Name                leases
  VG Name                ef720393-a513-4ce0-a5a7-43a3b2e04fc7
  LV UUID                coWhl8-pAfS-O059-13Ha-jken-m3gQ-evSkqu
  LV Write Access        read/write
  LV Creation host, time f20tre36i.localdomain, 2015-04-01 16:39:12 +0200
  LV Status              available
  # open                 0
  LV Size                2,00 GiB
  Current LE             16
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:9
   
  --- Logical volume ---
  LV Path                /dev/ef720393-a513-4ce0-a5a7-43a3b2e04fc7/ids
  LV Name                ids
  VG Name                ef720393-a513-4ce0-a5a7-43a3b2e04fc7
  LV UUID                gtT3jo-FXw2-Et3M-rCe5-4rTX-Ez7E-4ouWhJ
  LV Write Access        read/write
  LV Creation host, time f20tre36i.localdomain, 2015-04-01 16:39:13 +0200
  LV Status              available
  # open                 0
  LV Size                128,00 MiB
  Current LE             1
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:10
   
  --- Logical volume ---
  LV Path                /dev/ef720393-a513-4ce0-a5a7-43a3b2e04fc7/inbox
  LV Name                inbox
  VG Name                ef720393-a513-4ce0-a5a7-43a3b2e04fc7
  LV UUID                1yI0Ks-6Dan-6LQq-uQQY-ZbUU-ibcF-qJfUhP
  LV Write Access        read/write
  LV Creation host, time f20tre36i.localdomain, 2015-04-01 16:39:13 +0200
  LV Status              available
  # open                 0
  LV Size                128,00 MiB
  Current LE             1
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:11
   
  --- Logical volume ---
  LV Path                /dev/ef720393-a513-4ce0-a5a7-43a3b2e04fc7/master
  LV Name                master
  VG Name                ef720393-a513-4ce0-a5a7-43a3b2e04fc7
  LV UUID                DgGF1b-MWoX-ax15-l4Ge-U2P1-RiOL-mrnamt
  LV Write Access        read/write
  LV Creation host, time f20tre36i.localdomain, 2015-04-01 16:39:13 +0200
  LV Status              available
  # open                 0
  LV Size                1,00 GiB
  Current LE             8
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:12

Comment 4 Allon Mureinik 2015-05-20 12:58:58 UTC
You have to prepare the image (which will activate the LV) before you can write to it.

Comment 5 Sandro Bonazzola 2015-05-20 13:15:35 UTC
(In reply to Allon Mureinik from comment #4)
> You have to prepare the image (which will activate the LV) before you can
> write to it.

Never done that, it this something new?
do you have references to the verb to be used, and related parameters?

Comment 6 Simone Tiraboschi 2015-05-20 13:31:38 UTC
At that point we already did
        status = cli.createVolume(
            sdUUID,
            spUUID,
            imgUUID,
            str(
                int(
                    self.environment[ohostedcons.StorageEnv.IMAGE_SIZE_GB]
                ) * pow(2, 30)
            ),
            volFormat,
            preallocate,
            diskType,
            volUUID,
            self.environment[ohostedcons.StorageEnv.IMAGE_DESC],
        )

Which seams to successfully complete:
2015-05-20 13:10:27 INFO otopi.plugins.ovirt_hosted_engine_setup.vm.image image._misc:222 Creating VM Image
2015-05-20 13:10:27 DEBUG otopi.plugins.ovirt_hosted_engine_setup.vm.image image._misc:223 createVolume
2015-05-20 13:10:27 DEBUG otopi.plugins.ovirt_hosted_engine_setup.vm.image image._misc:249 {'status': {'message': 'OK', 'code': 0}, 'uuid': '8342e86f-9f58-492b-b1d9-d44ad12c2841'}
2015-05-20 13:10:27 DEBUG otopi.plugins.ovirt_hosted_engine_setup.vm.image image._misc:259 Created volume OK, request was:
- image: 117ccba3-b71a-4fc3-81c4-3ca03dde91f3
- volume: 62a44ba7-7a54-4289-b089-1776e8757b91
2015-05-20 13:10:27 DEBUG otopi.ovirt_hosted_engine_setup.tasks tasks.wait:50 Waiting for existing tasks to complete
2015-05-20 13:10:28 DEBUG otopi.ovirt_hosted_engine_setup.tasks tasks.wait:50 Waiting for existing tasks to complete
2015-05-20 13:10:28 DEBUG otopi.context context._executeMethod:141 Stage misc METHOD otopi.plugins.ovirt_hosted_engine_setup.vm.boot_disk.Plugin._misc
2015-05-20 13:10:28 DEBUG otopi.transaction transaction._prepare:80 preparing 'Image Transaction'


is it enough?

Comment 7 Allon Mureinik 2015-06-03 13:12:40 UTC
No - createVolume creates a volume on the storage (SPM verb). Preparing it exposes it (e.g., activates the lv) on the host (HSM verb). 
This is not new by any means.

Nir, can you work with Simone on this please?

Comment 8 Glen Taylor 2015-06-13 04:58:29 UTC
+1 Just hit the exact same roadblock with 3.5 RHEV deploying self-hosted engine to iSCSI target (on RHEL 7.1 host) with oVirt appliance ("disk" source).  Only difference was in the UUIDs.

From ovirt-hosted-engine-setup log:

2015-06-12 23:38:29 DEBUG otopi.plugins.ovirt_hosted_engine_setup.vm.boot_disk plugin.execute:866 execute-output: ('/bin/sudo', '-u', 'vdsm', '-g', 'kvm', '/bin/qemu-img', 'convert', '-O', 'raw', '/tmp/tmpBLclmN', '/rhev/data-center/mnt/blockSD/24393942-3006-454a-a378-642055049603/images/91e4da00-9d01-46e7-b3fd-356d01d38424/d8a8f43f-2029-48c2-b23a-f2088e5bd13f') stderr:
qemu-img: /rhev/data-center/mnt/blockSD/24393942-3006-454a-a378-642055049603/images/91e4da00-9d01-46e7-b3fd-356d01d38424/d8a8f43f-2029-48c2-b23a-f2088e5bd13f: error while converting raw: Could not create file: Permission denied

Comment 9 Elad 2015-08-13 07:56:29 UTC
The appliance image is transferred successfully using iSCSI storage. Hosted-engine deployment passes.

Used:
ovirt-3.6.0-5
ovirt-engine-appliance-20150802.0-1.el7.centos.noarch
ovirt-hosted-engine-setup-1.3.0-0.0.master.20150729070044.git26149d7.el7.noarch

Comment 10 Sandro Bonazzola 2015-11-04 13:43:57 UTC
oVirt 3.6.0 has been released on November 4th, 2015 and should fix this issue.
If problems still persist, please open a new BZ and reference this one.


Note You need to log in before you can comment on or make changes to this bug.