Bug 730297 - VM Lifecycle - VDSM must make sure that on setVmTicket currently connected user is disconnected
Summary: VM Lifecycle - VDSM must make sure that on setVmTicket currently connected us...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: vdsm
Version: 6.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Igor Lvovsky
QA Contact: Jakub Libosvar
URL:
Whiteboard:
Depends On: 707212 741533
Blocks: 689363 725466 818350 818351 818354
TreeView+ depends on / blocked
 
Reported: 2011-08-12 11:52 UTC by Barak
Modified: 2013-03-01 04:54 UTC (History)
9 users (show)

Fixed In Version: vdsm-4.9-97.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-12-06 07:36:49 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2011:1782 normal SHIPPED_LIVE new packages: vdsm 2011-12-06 11:55:51 UTC

Description Barak 2011-08-12 11:52:14 UTC
The problem is described in details in BZ#689363 comment 16.

Comment 2 Dan Kenigsberg 2011-08-15 06:01:49 UTC
This patch asks spice to disconnect current client when a new ticket is set

http://gerrit.usersys.redhat.com/814

Note that this does not eliminate the race between setTicket and login: the disconnect event may be lost, or the log off operation may fail. To solve it completely, we should have a synchronous logout operation, before (or during) spice connection.

Comment 3 Itamar Heim 2011-08-15 07:12:17 UTC
(In reply to comment #2)
> This patch asks spice to disconnect current client when a new ticket is set
> 
> http://gerrit.usersys.redhat.com/814
> 
> Note that this does not eliminate the race between setTicket and login: the
> disconnect event may be lost, or the log off operation may fail. To solve it
> completely, we should have a synchronous logout operation, before (or during)
> spice connection.

why logoff? just because users are connecting from another machine, doesn't mean we should kill (logoff) the session they had.
afaiu, normal behavior should be to lock the current session, not logoff it?
(unless we provide an action for the user to do this via a rhev-m permission)

Comment 4 Dan Kenigsberg 2011-08-15 14:16:32 UTC
(In reply to comment #3)
> why logoff? just because users are connecting from another machine, doesn't
> mean we should kill (logoff) the session they had.
> afaiu, normal behavior should be to lock the current session, not logoff it?
> (unless we provide an action for the user to do this via a rhev-m permission)

Sorry, my mixup. Synchronous session lock is needed to eliminate the race, logoff is too brutal.

Comment 6 Dan Kenigsberg 2011-09-04 09:24:29 UTC
Former patch has been reverted due to libvirt being unready back then. Reposting.

http://gerrit.usersys.redhat.com/893

Comment 7 Dan Kenigsberg 2011-09-25 15:05:48 UTC
QE: Verify that vdsm and spice honor the 4th argument to setVmTicket. It is either "keep", "disconnect" or "fail", meaning keep current client connection (even though it used an outdated password), disconnect it, or fail the password change.

RHEM currently uses only "disconnect".

Comment 8 Daniel Paikov 2011-09-26 13:43:26 UTC
On 4.9-104 this triggers very exotic exceptions in VDSM, which causes exceptions in RHEVM. Moving back to ASSIGNED.

Thread-16::ERROR::2011-09-26 19:13:47,135::guestIF::246::vm.Vm::(run) vmId=`669247c4-4fb4-476a-9bde-4c38ba9ea155`::Run exception: ^@^@^@^A^@
^@^@^C^@^@^@^S^@^@^@^C397^@^@^@^A^@^@^@^C^@^@^@^S^@^@^@^C396^@^@^@^A^@^@^@^C^@^@^@^S^@^@^@^C396^@^@^@^A^@^@^@^C^@^@^@^S^@^@^@^C397^@^@^@^A^@
^@^@^C^@^@^@^S^@^@^@^C397^@^@^@^A^@^@^@^C^@^@^@^S^@^@^@^C398^@^@^@^A^@^@^@^C^@^@^@^S^@^@^@^C398^@^@^@^A^@^@^@^C^@^@^@^S^@^@^@^C398^@^@^@^A^@
^@^@^C^@^@^@^S^@^@^@^C398^@^@^@^A^@^@^@^C^@^@^@^S^@^@^@^C398^@^@^@^A^@^@^@^C^@^@^@^S^@^@^@^C398^@^@^@^A^@^@^@^C^@^@^@^S^@^@^@^C398^@^@^@^A^@
^@^@^C^@^@^@^S^@^@^@^C398^@^@^@^A^@^@^@^C^@^@^@&^@^@^@^HPaikov@qa.qumranet.com^@^@^@^A^@^@^@^C^@^@^@^P^@^@^@^O^@^@^@^A^@^@^@^C^@^@^@^S^@^@^@
^C384^@^@^@^A^@^@^@^C^@^@^@^S^@^@^@^C385^@^@^@^A^@^@^@^C^@^@^@^S^@^@^@^C386^@^@^@^A^@^@^@^C^@^@^@^S^@^@^@^C383^@^@^@^A^@^@^@^C^@^@^@^S^@^@^@
^C381^@^@^@^A^@^@^@^C^@^@^@^S^@^@^@^C381^@^@^@^A^@^@^@^C^@^@^@^S^@^@^@^C381^@^@^@^A^@^@^@^C^@^@^@^S^@^@^@^C381^@^@^@^A^@^@^@^C^@^@^@^S^@^@^@
^C381^@^@^@^A^@^@^@^C^@^@^@^P^@^@^@
Traceback (most recent call last):
  File "/usr/share/vdsm/guestIF.py", line 242, in run
    (message, args) = self._parseLine(line)
  File "/usr/share/vdsm/guestIF.py", line 223, in _parseLine
    args = json.loads(line.decode('utf8'))
  File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.6/json/decoder.py", line 338, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Comment 9 Gal Hammer 2011-09-26 14:33:47 UTC
Which version of the RHEV-agent is running on the VM?

Comment 10 Daniel Paikov 2011-09-26 14:38:24 UTC
(In reply to comment #9)
> Which version of the RHEV-agent is running on the VM?

RHEV-toolSetup-3.0-20. Which should we be running?

Comment 11 Gal Hammer 2011-09-26 14:46:40 UTC
I would guess that either 21 or 22.

Comment 12 Dan Kenigsberg 2011-09-27 07:11:07 UTC
Daniel, yeah that's ugly (I'm not suggesting that exotic == ugly!). As Gal said, you should be verifying this bug with new guest. HOWEVER, there might be material here for another bug, about vdsm's handling of bad guest data. Please explain how an exception was triggered in RHEVM on a new bug.

Comment 13 Daniel Paikov 2011-09-27 07:35:19 UTC
* I didn't find 22, but I tried this on 21 and it looks even worse. Trying to open 2 SPICE sessions to the same VM causes libvirt to die and VDSM to restart itself. I don't understand why this bug is back on ON_QA if nobody verified that upgrading to 21/22 makes any (positive) difference.

* How an exception was triggered? By trying to verify this BZ - opening 2 SPICE sessions either from admin portal+user portal, or from 2 user portal sessions, etc.

Thread-76::ERROR::2011-09-27 13:24:38,932::libvirtconnection::73::vds::(wrapper) connection to libvirt broken. taking vdsm down.
MainThread::INFO::2011-09-27 13:24:38,933::vdsm::79::vds::(run) <WorkerThread(Thread-7, started 140304929765120)>
Thread-76::DEBUG::2011-09-27 13:24:38,933::clientIF::142::vds::(prepareForShutdown) cannot run prepareForShutdown concurrently
MainThread::INFO::2011-09-27 13:24:38,933::vdsm::79::vds::(run) <WorkerThread(Thread-4, started 140304961234688)>
MainThread::INFO::2011-09-27 13:24:38,934::vdsm::79::vds::(run) <WorkerThread(Thread-6, started 140304940254976)>
MainThread::INFO::2011-09-27 13:24:38,935::vdsm::79::vds::(run) <WorkerThread(Thread-5, started 140304950744832)>
MainThread::INFO::2011-09-27 13:24:38,935::vdsm::79::vds::(run) <WorkerThread(Thread-10, started 140304558581504)>
MainThread::INFO::2011-09-27 13:24:38,936::vdsm::79::vds::(run) <KsmMonitorThread(KsmMonitor, started daemon 140304527111936)>
Thread-76::ERROR::2011-09-27 13:24:38,936::clientIF::62::vds::(wrapper) Traceback (most recent call last):
  File "/usr/share/vdsm/clientIF.py", line 58, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/clientIF.py", line 424, in setVmTicket
    return vm.setTicket(otp, seconds, connAct)
  File "/usr/share/vdsm/libvirtvm.py", line 1082, in setTicket
    graphics = xml.dom.minidom.parseString(self._dom.XMLDesc(0)) \
  File "/usr/share/vdsm/libvirtvm.py", line 328, in f
    ret = attr(*args, **kwargs)
  File "/usr/share/vdsm/libvirtconnection.py", line 63, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 381, in XMLDesc
    if ret is None: raise libvirtError ('virDomainGetXMLDesc() failed', dom=self)
libvirtError: Cannot write data: Broken pipe
Thread-15::DEBUG::2011-09-27 13:24:39,790::utils::345::vm.Vm::(run) vmId=`669247c4-4fb4-476a-9bde-4c38ba9ea155`::Stats thread finished
MainThread::INFO::2011-09-27 13:24:42,818::vdsm::71::vds::(run) I am the actual vdsm 4.9-104

Comment 14 Daniel Paikov 2011-09-27 08:19:57 UTC
Added dependency on libvirt bug #741533.

Comment 15 Jakub Libosvar 2011-10-06 06:17:19 UTC
According comment https://bugzilla.redhat.com/show_bug.cgi?id=707212#c20 moving to verified - vdsm-106

Comment 16 errata-xmlrpc 2011-12-06 07:36:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2011-1782.html


Note You need to log in before you can comment on or make changes to this bug.