Bug 1293299

Summary: console does not work after reverting to a snapshot created in RHEV 3.5
Product: Red Hat Enterprise Virtualization Manager Reporter: Petr Spacek <pspacek>
Component: ovirt-engineAssignee: Shmuel Melamud <smelamud>
Status: CLOSED WONTFIX QA Contact: Israel Pinto <ipinto>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.6.1CC: ahadas, didi, fromani, gklein, ipinto, lsurette, mavital, mgoldboi, michal.skrivanek, rbalakri, Rhev-m-bugs, sbonazzo, tjelinek, yeylon, ykaul
Target Milestone: ---Keywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-02-08 13:14:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1260177    
Bug Blocks: 1213937, 1285700    
Attachments:
Description Flags
logs
none
vms_xml none

Description Petr Spacek 2015-12-21 11:20:41 UTC
Description of problem:
This started to happen after upgrade to RHEV-M 3.6.1.

When I revert a VM to a snapshot created before the upgrade (originally on 3.5.z), the VM does not work properly after start.

The "Console" button in "User portal" is grayed out. "Console options" dialog has all the options grayed out, too. 

Version-Release number of selected component (if applicable):
3.6.1-0.2.el6

How reproducible:
100 %

Steps to Reproduce:
0. Start with RHEV 3.5.5
1. Create snapshot of a VM configured with SPICE console
2. Upgrade to RHEV 3.6.1
3. Revert VM to the snapshot
4. Attempt to open VM console

Actual results:
The console does not work.

Expected results:
The console works.

Additional info:
* Admin portal shows "None" in column "Graphics"

* Admin portal shows following line on "VM devides" tab for the VM:
qxl {slot=0x02, bus=0x00, domain=0x0000, type=pci, function=0x0} {vram=32768, heads=1}

Comment 2 Arik 2016-01-07 13:50:46 UTC
*** Bug 1293135 has been marked as a duplicate of this bug. ***

Comment 3 Shmuel Melamud 2016-01-20 10:44:18 UTC
*** Bug 1298509 has been marked as a duplicate of this bug. ***

Comment 4 Israel Pinto 2016-02-08 08:51:35 UTC
Verify on:
3.6: RHEVM Version: 3.6.3-0.1.el6 
3.5: RHEVM:3.5.7-0.1.el6ev 

Steps to Reproduce:
1. On rhevm 3.5: Create snapshot of a VM configured with SPICE console 
(vm: vm-user-portal) 
2. Upgrade to RHEV 3.6.3
3. Revert VM to the snapshot after upgrade
4. Attempt to open VM console

Results:
1. The "Console" button in "User portal" is NOT grayed out. 
And "Console options" dialog options NOT grayed out.
2. Can't open console, got error:
User admin@internal failed to initiate a console session for VM vm-user-portal

Notes:
1. It happened also with VM that created on Admin portal (with the same scenario, vm: vm_webadmin) 
2. I also check it with connection to Web portal with FQDN and not IP

Adding logs

From engine_log:
2016-02-08 09:45:07,896 WARN  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp-/127.0.0.1:8702-6) [1ec45e4d] The message key 'VmLogon' is missing from 'bundles/ExecutionMessages'
2016-02-08 09:45:07,947 INFO  [org.ovirt.engine.core.bll.VmLogonCommand] (ajp-/127.0.0.1:8702-6) [1ec45e4d] Running command: VmLogonCommand internal: false. Entities affected :  ID: efefe9b0-4bf8-4fc1-acf7-89f1fea94e17 Type: VMAction group CONNECT_TO_VM with role type USER
2016-02-08 09:45:07,966 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.VmLogonVDSCommand] (ajp-/127.0.0.1:8702-6) [1ec45e4d] START, VmLogonVDSCommand(HostName = dhcp163-73.scl.lab.tlv.redhat.com, VmLogonVDSCommandParameters:{runAsync='true', hostId='7e2c5efa-5df0-4b36-a23c-d195734aabde', vmId='efefe9b0-4bf8-4fc1-acf7-89f1fea94e17', domain='internal', password='***', userName='admin@internal'}), log id: 7e23dba5
2016-02-08 09:45:07,970 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VmLogonVDSCommand] (ajp-/127.0.0.1:8702-6) [1ec45e4d] Failed in 'VmLogonVDS' method
2016-02-08 09:45:07,980 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [1ec45e4d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM dhcp163-73.scl.lab.tlv.redhat.com command failed: Guest agent non-responsive
2016-02-08 09:45:07,980 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.VmLogonVDSCommand] (ajp-/127.0.0.1:8702-6) [1ec45e4d] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.VmLogonVDSCommand' return value 'StatusOnlyReturnForXmlRpc [status=StatusForXmlRpc [code=19, message=Guest agent non-responsive]]'
2016-02-08 09:45:07,981 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.VmLogonVDSCommand] (ajp-/127.0.0.1:8702-6) [1ec45e4d] HostName = dhcp163-73.scl.lab.tlv.redhat.com
2016-02-08 09:45:07,981 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VmLogonVDSCommand] (ajp-/127.0.0.1:8702-6) [1ec45e4d] Command 'VmLogonVDSCommand(HostName = dhcp163-73.scl.lab.tlv.redhat.com, VmLogonVDSCommandParameters:{runAsync='true', hostId='7e2c5efa-5df0-4b36-a23c-d195734aabde', vmId='efefe9b0-4bf8-4fc1-acf7-89f1fea94e17', domain='internal', password='***', userName='admin@internal'})' execution failed: VDSGenericException: VDSErrorException: Failed to VmLogonVDS, error = Guest agent non-responsive, code = 19
2016-02-08 09:45:07,981 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.VmLogonVDSCommand] (ajp-/127.0.0.1:8702-6) [1ec45e4d] FINISH, VmLogonVDSCommand, log id: 7e23dba5
2016-02-08 09:45:07,981 ERROR [org.ovirt.engine.core.bll.VmLogonCommand] (ajp-/127.0.0.1:8702-6) [1ec45e4d] Command 'org.ovirt.engine.core.bll.VmLogonCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to VmLogonVDS, error = Guest agent non-responsive, code = 19 (Failed with error nonresp and code 19)
2016-02-08 09:45:07,995 ERROR [org.ovirt.engine.core.bll.VmLogonCommand] (ajp-/127.0.0.1:8702-6) [1ec45e4d] Transaction rolled-back for command 'org.ovirt.engine.core.bll.VmLogonCommand'.
2016-02-08 09:45:10,448 INFO  [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-4) [5413c69a] Running command: SetVmTicketCommand internal: true. Entities affected :  ID: efefe9b0-4bf8-4fc1-acf7-89f1fea94e17 Type: VMAction group CONNECT_TO_VM with role type USER
2016-02-08 09:45:10,475 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-4) [5413c69a] START, SetVmTicketVDSCommand(HostName = dhcp163-73.scl.lab.tlv.redhat.com, SetVmTicketVDSCommandParameters:{runAsync='true', hostId='7e2c5efa-5df0-4b36-a23c-d195734aabde', vmId='efefe9b0-4bf8-4fc1-acf7-89f1fea94e17', protocol='SPICE', ticket='RRXQ5hLBwToA', validTime='120', userName='admin', userId='fdfc627c-d875-11e0-90f0-83df133b58cc', disconnectAction='LOCK_SCREEN'}), log id: a95c88c
2016-02-08 09:45:10,502 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-4) [5413c69a] Unexpected return value: StatusForXmlRpc [code=-32603, message=internal error unable to execute QEMU command '__com.redhat_set_password': Could not set password]
2016-02-08 09:45:10,502 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-4) [5413c69a] Failed in 'SetVmTicketVDS' method
2016-02-08 09:45:10,508 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [5413c69a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM dhcp163-73.scl.lab.tlv.redhat.com command failed: internal error unable to execute QEMU command '__com.redhat_set_password': Could not set password
2016-02-08 09:45:10,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-4) [5413c69a] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand' return value 'StatusOnlyReturnForXmlRpc [status=StatusForXmlRpc [code=-32603, message=internal error unable to execute QEMU command '__com.redhat_set_password': Could not set password]]'
2016-02-08 09:45:10,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-4) [5413c69a] HostName = dhcp163-73.scl.lab.tlv.redhat.com
2016-02-08 09:45:10,509 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-4) [5413c69a] Command 'SetVmTicketVDSCommand(HostName = dhcp163-73.scl.lab.tlv.redhat.com, SetVmTicketVDSCommandParameters:{runAsync='true', hostId='7e2c5efa-5df0-4b36-a23c-d195734aabde', vmId='efefe9b0-4bf8-4fc1-acf7-89f1fea94e17', protocol='SPICE', ticket='RRXQ5hLBwToA', validTime='120', userName='admin', userId='fdfc627c-d875-11e0-90f0-83df133b58cc', disconnectAction='LOCK_SCREEN'})' execution failed: VDSGenericException: VDSErrorException: Failed to SetVmTicketVDS, error = internal error unable to execute QEMU command '__com.redhat_set_password': Could not set password, code = -32603
2016-02-08 09:45:10,509 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-4) [5413c69a] FINISH, SetVmTicketVDSCommand, log id: a95c88c
2016-02-08 09:45:10,509 ERROR [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-4) [5413c69a] Command 'org.ovirt.engine.core.bll.SetVmTicketCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to SetVmTicketVDS, error = internal error unable to execute QEMU command '__com.redhat_set_password': Could not set password, code = -32603 (Failed with error unexpected and code 16)
2016-02-08 09:45:10,518 ERROR [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-4) [5413c69a] Transaction rolled-back for command 'org.ovirt.engine.core.bll.SetVmTicketCommand'.
2016-02-08 09:45:10,535 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [5413c69a] Correlation ID: 5413c69a, Call Stack: null, Custom Event ID: -1, Message: User admin@internal failed to initiate a console session for VM vm-user-portal

Comment 5 Israel Pinto 2016-02-08 08:52:14 UTC
Created attachment 1122085 [details]
logs

Comment 6 Israel Pinto 2016-02-08 08:52:45 UTC
Created attachment 1122086 [details]
vms_xml

Comment 8 Israel Pinto 2016-02-08 11:21:25 UTC
Host 3.5 
OS Version: RHEL - 6Server - 6.7.0.3.el6
Kernel Version: 2.6.32 - 573.12.1.el6.x86_64
KVM Version: 0.12.1.2 - 2.479.el6_7.4
LIBVIRT Version: libvirt-0.10.2-54.el6_7.3
VDSM Version: vdsm-4.16.33-1.el6ev
SPICE Version: 0.12.4 - 12.el6_7.3

Comment 10 Israel Pinto 2016-02-08 12:31:27 UTC
all in the attached files

Comment 11 Michal Skrivanek 2016-02-08 13:14:43 UTC
the locking implemented by bug 1292398 is preventing you to restore/resume the snapshot/suspended VM, therefore this situation can no longer happen

I believe the behavior in 3.5 compatibility level in 3.6 was verified recently, so we should be good...