Descriptionvvyazmin@redhat.com
2013-02-25 17:06:03 UTC
Created attachment 702434[details]
## Logs PythonSDK
Description of problem: Power on VM's action stuck after 110 VM's
Version-Release number of selected component (if applicable):
RHEVM 3.2 - SF08 environment:
RHEVM: rhevm-3.2.0-9.el6ev.noarch
VDSM: vdsm-4.10.2-9.0.el6ev.x86_64
LIBVIRT: libvirt-0.10.2-18.el6.x86_64
QEMU & KVM: qemu-kvm-rhev-0.12.1.2-2.355.el6_4.1.x86_64
SANLOCK: sanlock-2.6-2.el6.x86_64
How reproducible:
100%
No problems found in RHEVM 3.2 - SF07
Steps to Reproduce:
1. power on 250 VM's in loop (see function below)
Actual results:
* And after approximately 110 power on VM's action , and session stuck:
Expected results:
Succeed power on multiple VM's via PythonSDK
Additional info:
### Start VM ###
def startVm(VM_NAME, VM_NAME_FROM, VM_NAME_TO):
timestamp_SCALE()
for index in range(VM_NAME_FROM, VM_NAME_TO+1):
real_VM_NAME = VM_NAME+str(index)
# print real_VM_NAME
try:
if api.vms.get(real_VM_NAME).status.state != 'up':
print 'Starting VM: ' + real_VM_NAME
print 'fetching vm:' + real_VM_NAME
vm = api.vms.get(real_VM_NAME)
print 'starting vm:' + vm.id
print '====================='
vm.start()
print 'Waiting for VM to reach Up status'
while api.vms.get(VM_NAME).status.state != 'up':
sleep(1)
else:
print 'VM: ' + real_VM_NAME + ' already up'
except Exception as e:
print 'Failed to Start VM:\n%s' % str(e)
timestamp_SCALE()
startVm('pool_01-10vms-', 1, 250)
Stuck on search action
send: 'GET /api/vms?search=name%3Dpool_01-10vms-111 HTTP/1.1\r\nHost: puma31.scl.lab.tlv.redhat.com\r\nAccept-Encoding: identity\r\nFilter: False\r\nPrefer: persistent-auth\r\ncookie: JSESSIONID=okcOm9wsWDlIsttRLclXd5QF\r\nContent-type: application/xml\r\nAuthorization: Basic YWRtaW5AaW50ZXJuYWw6cXVtNW5ldA==\r\n\r\n'
/var/log/ovirt-engine/engine.log
/var/log/vdsm/vdsm.log
Comment 2vvyazmin@redhat.com
2013-02-25 20:49:12 UTC
Same problem I get after deleting in loop VM's. Delete action stuck after 110 VM's.
Comment 8vvyazmin@redhat.com
2013-02-26 12:50:38 UTC
Add logs,
Add regression
Comment 9Michael Pasternak
2013-02-27 06:25:57 UTC
(In reply to comment #2)
> Same problem I get after deleting in loop VM's. Delete action stuck after
> 110 VM's.
This is not related to the action nor to delete, you did
api.vms.get(real_VM_NAME).action() combining two operations (get, action) in
in two one, to identify the problem we split it in two separate operation (see copy/paste you did in your bug description) and api got stuck at
api.vms.get(real_VM_NAME) that uses 'search'.
I saw a problem.
The issue was solved after restart of ovirt machine.
Due to thread dump - ovirt was stacked inside java internal classes.
My advice after upgrade perform restart of ovirt machine.
As agreed, closing bug as worked for me
Created attachment 702434 [details] ## Logs PythonSDK Description of problem: Power on VM's action stuck after 110 VM's Version-Release number of selected component (if applicable): RHEVM 3.2 - SF08 environment: RHEVM: rhevm-3.2.0-9.el6ev.noarch VDSM: vdsm-4.10.2-9.0.el6ev.x86_64 LIBVIRT: libvirt-0.10.2-18.el6.x86_64 QEMU & KVM: qemu-kvm-rhev-0.12.1.2-2.355.el6_4.1.x86_64 SANLOCK: sanlock-2.6-2.el6.x86_64 How reproducible: 100% No problems found in RHEVM 3.2 - SF07 Steps to Reproduce: 1. power on 250 VM's in loop (see function below) Actual results: * And after approximately 110 power on VM's action , and session stuck: Expected results: Succeed power on multiple VM's via PythonSDK Additional info: ### Start VM ### def startVm(VM_NAME, VM_NAME_FROM, VM_NAME_TO): timestamp_SCALE() for index in range(VM_NAME_FROM, VM_NAME_TO+1): real_VM_NAME = VM_NAME+str(index) # print real_VM_NAME try: if api.vms.get(real_VM_NAME).status.state != 'up': print 'Starting VM: ' + real_VM_NAME print 'fetching vm:' + real_VM_NAME vm = api.vms.get(real_VM_NAME) print 'starting vm:' + vm.id print '=====================' vm.start() print 'Waiting for VM to reach Up status' while api.vms.get(VM_NAME).status.state != 'up': sleep(1) else: print 'VM: ' + real_VM_NAME + ' already up' except Exception as e: print 'Failed to Start VM:\n%s' % str(e) timestamp_SCALE() startVm('pool_01-10vms-', 1, 250) Stuck on search action send: 'GET /api/vms?search=name%3Dpool_01-10vms-111 HTTP/1.1\r\nHost: puma31.scl.lab.tlv.redhat.com\r\nAccept-Encoding: identity\r\nFilter: False\r\nPrefer: persistent-auth\r\ncookie: JSESSIONID=okcOm9wsWDlIsttRLclXd5QF\r\nContent-type: application/xml\r\nAuthorization: Basic YWRtaW5AaW50ZXJuYWw6cXVtNW5ldA==\r\n\r\n' /var/log/ovirt-engine/engine.log /var/log/vdsm/vdsm.log