Bug 1149714 - Failed to create VM based on [rhos-qe-dept] SHELLSHOCK - rhel-guest-image-6.5 (6.5-20140930.0) imported template on RHEL6.6.
Summary: Failed to create VM based on [rhos-qe-dept] SHELLSHOCK - rhel-guest-image-6.5...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.5.0
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: ---
: 3.5.0
Assignee: Nobody
QA Contact:
URL:
Whiteboard: virt
Depends On:
Blocks: rhev35betablocker rhev35rcblocker rhev35gablocker
TreeView+ depends on / blocked
 
Reported: 2014-10-06 14:01 UTC by Nikolai Sednev
Modified: 2015-08-16 08:28 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-10-19 13:26:35 UTC
oVirt Team: ---
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
errormessagescreenshot (198.60 KB, image/png)
2014-10-06 14:01 UTC, Nikolai Sednev
no flags Details
logs (389.67 KB, application/x-gzip)
2014-10-06 14:15 UTC, Nikolai Sednev
no flags Details
newestlogs (1.61 MB, application/x-gzip)
2014-10-08 09:00 UTC, Nikolai Sednev
no flags Details
logs from 12_10_14 (1.77 MB, application/x-gzip)
2014-10-12 05:55 UTC, Nikolai Sednev
no flags Details

Description Nikolai Sednev 2014-10-06 14:01:36 UTC
Created attachment 944266 [details]
errormessagescreenshot

Description of problem:
Failed to create VM based on [rhos-qe-dept] SHELLSHOCK - rhel-guest-image-6.5 (6.5-20140930.0) imported template on RHEL6.6.

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-0.12.1.2-2.448.el6.x86_64
vdsm-4.16.5-2.el6ev.x86_64
sanlock-2.8-1.el6.x86_64
libvirt-0.10.2-46.el6.x86_64
2.6.32-502.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Take guest image and export via iso-uploader.
2.Import template from export domain. 
3.Create VM from template and stry to run it on RHEL6.6 host.

Actual results:
Error received and VM not starting.

Expected results:
VM should be created from imported guest image template and to start properly over RHEL6.6.

Additional info:

Comment 1 Nikolai Sednev 2014-10-06 14:15:30 UTC
Created attachment 944272 [details]
logs

Comment 2 Michal Skrivanek 2014-10-07 06:51:42 UTC
Allon, can someone from storage check this?
During the VM creation I can see 

Message: VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = low level Image copy failed, code = 261

at 2014-10-06 15:53:10,334 in the engine.log.

Nikolai, I suppose the vdsm logs are not the right ones, very narrow time window is captured. We need to see the VM creation (storage task)

Comment 3 Allon Mureinik 2014-10-07 17:56:34 UTC
(In reply to Michal Skrivanek from comment #2)
> Allon, can someone from storage check this?
> During the VM creation I can see 
> 
> Message: VDSGenericException: VDSErrorException: Failed to
> HSMGetAllTasksStatusesVDS, error = low level Image copy failed, code = 261
Sure.
Tal - can you take a look please once the logs are provided:

> at 2014-10-06 15:53:10,334 in the engine.log.
> 
> Nikolai, I suppose the vdsm logs are not the right ones, very narrow time
> window is captured. We need to see the VM creation (storage task)

Comment 4 Nikolai Sednev 2014-10-08 09:00:07 UTC
I tried to reproduce on vt5,VM failed to start.
VM started on RHEL7 host instead of RHEL6.6, then migration failed to RHEL6.6 because selinux bug.
Migration fails because of the selinux is enforcing, while guest-agent crush I have no idea.
vdsm-4.16.6-1.el6ev.x86_64
qemu-kvm-rhev-0.12.1.2-2.448.el6.x86_64
sanlock-2.8-1.el6.x86_64
libvirt-0.10.2-46.el6.x86_64
rhevm-3.5.0-0.14.beta.el6ev.noarch

New logs attached.

Comment 5 Nikolai Sednev 2014-10-08 09:00:57 UTC
Created attachment 944886 [details]
newestlogs

Comment 6 Wei Shi 2014-10-08 10:09:53 UTC
I cannot reproduce this bug on:
  rhevm-3.5.0-0.14.beta.el6ev.noarch
  vdsm-4.14.7-3.el6ev.x86_64
  qemu-kvm-rhev-0.12.1.2-2.415.el6_5.3.x86_64

I use rhevm-image-uploader instead of iso-uploader based on the steps in TCMS.

1. Upload ovf with rhevm-image-uploader
2. Import template from export domain. 
3. Create VM from template and run it on RHEL-6.6-20140926.0 host.

I can login to rhel-guest-image-6.5 (6.5-20140930.0) successfully through virt-viewer console.

Comment 7 Michal Skrivanek 2014-10-10 17:16:41 UTC
also, needinfo on Tal still applicable?

Comment 8 Nikolai Sednev 2014-10-12 05:53:34 UTC
(In reply to Wei Shi from comment #6)
> I cannot reproduce this bug on:
>   rhevm-3.5.0-0.14.beta.el6ev.noarch
>   vdsm-4.14.7-3.el6ev.x86_64
>   qemu-kvm-rhev-0.12.1.2-2.415.el6_5.3.x86_64
> 
> I use rhevm-image-uploader instead of iso-uploader based on the steps in
> TCMS.
> 
> 1. Upload ovf with rhevm-image-uploader
> 2. Import template from export domain. 
> 3. Create VM from template and run it on RHEL-6.6-20140926.0 host.
> 
> I can login to rhel-guest-image-6.5 (6.5-20140930.0) successfully through
> virt-viewer console.

Successfully reproduced again on my setup with the error message received as within the attachment, logs attached as well.

Components:
libvirt-0.10.2-46.el6.x86_64
vdsm-4.16.6-1.el6ev.x86_64
qemu-kvm-rhev-0.12.1.2-2.448.el6.x86_64
sanlock-2.8-1.el6.x86_64
libvirt-0.10.2-29.el6_5.9.x86_64

Comment 9 Nikolai Sednev 2014-10-12 05:55:20 UTC
Created attachment 946072 [details]
logs from 12_10_14

Comment 10 Michal Skrivanek 2014-10-12 06:27:16 UTC
(In reply to Nikolai Sednev from comment #4)
> VM started on RHEL7 host instead of RHEL6.6, then migration failed to
avoid that. Not supported and cause random issues

Comment 11 Michal Skrivanek 2014-10-12 07:07:33 UTC
(In reply to Nikolai Sednev from comment #9)
sorry, again can't find relevant in the logs other than lot of noise about network connectivity problems in engine.log
Please try to pinpoint the error you see, describe exactly the flow

Comment 12 Nikolai Sednev 2014-10-13 09:26:52 UTC
Flow is as described in description of a bug:
Steps to Reproduce:
1.Take guest image and export via iso-uploader.
2.Import template from export domain. 
3.Create VM from template and try to run it on RHEL6.6 host.

I don't think I can provide more info than this, logs attached and screenshot as well, sorry, but that's my best to describe. If you want to see this happens, please contact me directly and observe on my setup.

Comment 13 yuliu 2014-10-13 10:42:48 UTC
(In reply to Nikolai Sednev from comment #12)
> Flow is as described in description of a bug:
> Steps to Reproduce:
> 1.Take guest image and export via iso-uploader.
> 2.Import template from export domain. 
> 3.Create VM from template and try to run it on RHEL6.6 host.
> 
> I don't think I can provide more info than this, logs attached and
> screenshot as well, sorry, but that's my best to describe. If you want to
> see this happens, please contact me directly and observe on my setup.

Did you try upload the image by engine-image-upload ? Since it's a tar image not an ISO file.

I can't reproduce this bug either, needinfo Ryan for help.

Comment 14 Nikolai Sednev 2014-10-13 12:10:39 UTC
(In reply to yuliu from comment #13)
> (In reply to Nikolai Sednev from comment #12)
> > Flow is as described in description of a bug:
> > Steps to Reproduce:
> > 1.Take guest image and export via iso-uploader.
> > 2.Import template from export domain. 
> > 3.Create VM from template and try to run it on RHEL6.6 host.
> > 
> > I don't think I can provide more info than this, logs attached and
> > screenshot as well, sorry, but that's my best to describe. If you want to
> > see this happens, please contact me directly and observe on my setup.
> 
> Did you try upload the image by engine-image-upload ? Since it's a tar image
> not an ISO file.
> 
> I can't reproduce this bug either, needinfo Ryan for help.

I did only as described above, only ISO, upload is OK, import then is fine, VM creation is perfect, but VM start failed.

Comment 15 Fabian Deutsch 2014-10-13 12:23:25 UTC
(In reply to Nikolai Sednev from comment #14)
> (In reply to yuliu from comment #13)
> > (In reply to Nikolai Sednev from comment #12)
> > > Flow is as described in description of a bug:
> > > Steps to Reproduce:
> > > 1.Take guest image and export via iso-uploader.
> > > 2.Import template from export domain. 
> > > 3.Create VM from template and try to run it on RHEL6.6 host.
> > > 
> > > I don't think I can provide more info than this, logs attached and
> > > screenshot as well, sorry, but that's my best to describe. If you want to
> > > see this happens, please contact me directly and observe on my setup.
> > 
> > Did you try upload the image by engine-image-upload ? Since it's a tar image
> > not an ISO file.
> > 
> > I can't reproduce this bug either, needinfo Ryan for help.
> 
> I did only as described above, only ISO, upload is OK, import then is fine,
> VM creation is perfect, but VM start failed.

Nikolai, does the VM come up at all? Or is not even the "bios" shown?

Comment 16 Fabian Deutsch 2014-10-13 12:26:54 UTC
According to comment 4 and comment 12 this can only be reproduce on RHEL 6.6.
According to comment 4 this can not be reproduce on RHEL 7.

So to me this looks like a bug of RHEV on RHEL 6.6, and for me this is not a reason to block the RHEL guest image.

We can not make RHEV 3.5 (which is in beta) a blocker for an image with security fixes.

Comment 17 Nikolai Sednev 2014-10-13 12:34:04 UTC
(In reply to Fabian Deutsch from comment #15)
> (In reply to Nikolai Sednev from comment #14)
> > (In reply to yuliu from comment #13)
> > > (In reply to Nikolai Sednev from comment #12)
> > > > Flow is as described in description of a bug:
> > > > Steps to Reproduce:
> > > > 1.Take guest image and export via iso-uploader.
> > > > 2.Import template from export domain. 
> > > > 3.Create VM from template and try to run it on RHEL6.6 host.
> > > > 
> > > > I don't think I can provide more info than this, logs attached and
> > > > screenshot as well, sorry, but that's my best to describe. If you want to
> > > > see this happens, please contact me directly and observe on my setup.
> > > 
> > > Did you try upload the image by engine-image-upload ? Since it's a tar image
> > > not an ISO file.
> > > 
> > > I can't reproduce this bug either, needinfo Ryan for help.
> > 
> > I did only as described above, only ISO, upload is OK, import then is fine,
> > VM creation is perfect, but VM start failed.
> 
> Nikolai, does the VM come up at all? Or is not even the "bios" shown?

VM not started at all. I didn't saw SPICE coming available for pressing it to see the BIOS.

Comment 19 Michal Skrivanek 2014-10-13 12:53:56 UTC
finally found

2014-10-12 08:47:31,491 ERROR [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (ajp-/127.0.0.1:8702-3) [df52a98] Command org.ovirt.engine.core.bll
.network.vm.ActivateDeactivateVmNicCommand throw exception: java.lang.NullPointerException
        at org.ovirt.engine.core.bll.utils.VmDeviceUtils.updateBootOrderInVmDevice(VmDeviceUtils.java:641) [bll.jar:]

in the noise of engine.log

Comment 20 Fabian Deutsch 2014-10-13 14:49:31 UTC
(In reply to Nikolai Sednev from comment #17)
> (In reply to Fabian Deutsch from comment #15)
> > (In reply to Nikolai Sednev from comment #14)
> > > (In reply to yuliu from comment #13)
> > > > (In reply to Nikolai Sednev from comment #12)
> > > > > Flow is as described in description of a bug:
> > > > > Steps to Reproduce:
> > > > > 1.Take guest image and export via iso-uploader.
> > > > > 2.Import template from export domain. 
> > > > > 3.Create VM from template and try to run it on RHEL6.6 host.
> > > > > 
> > > > > I don't think I can provide more info than this, logs attached and
> > > > > screenshot as well, sorry, but that's my best to describe. If you want to
> > > > > see this happens, please contact me directly and observe on my setup.
> > > > 
> > > > Did you try upload the image by engine-image-upload ? Since it's a tar image
> > > > not an ISO file.
> > > > 
> > > > I can't reproduce this bug either, needinfo Ryan for help.
> > > 
> > > I did only as described above, only ISO, upload is OK, import then is fine,
> > > VM creation is perfect, but VM start failed.
> > 
> > Nikolai, does the VM come up at all? Or is not even the "bios" shown?
> 
> VM not started at all. I didn't saw SPICE coming available for pressing it
> to see the BIOS.

Okay, then this is very very likely not caused by the image itself.

Comment 21 Michal Skrivanek 2014-10-13 20:00:30 UTC
(In reply to Nikolai Sednev from comment #14)
> I did only as described above, only ISO, upload is OK, import then is fine,
> VM creation is perfect, but VM start failed.

you say VM creation succeeded? Then it's a different thing than what you reported before since the screenshot attached here says the VM creation from template failed.

Comment 24 Omer Frenkel 2014-10-19 13:26:35 UTC
looking at the log, there are bunch of errors, most of them transient, some for network and some creations failed with "low level Image copy failed, code = 261"

tried to reproduce together with Nikolai, we couldn't see it again.
so closing, please re-open if it reproduce.


Note You need to log in before you can comment on or make changes to this bug.