Bug 1088805
| Summary: | Appear vdsm-tool exception in time of vdsm package upgrade from 3.3 to 3.4 | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Virtualization Manager | Reporter: | Artyom <alukiano> | ||||
| Component: | vdsm | Assignee: | Yaniv Bronhaim <ybronhei> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Artyom <alukiano> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | unspecified | CC: | aberezin, bazulay, eedri, fsimonce, gklein, iheim, kgoldbla, lbopf, lpeer, oourfali, pstehlik, yeylon | ||||
| Target Milestone: | --- | Keywords: | ZStream | ||||
| Target Release: | 3.4.2 | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | infra | ||||||
| Fixed In Version: | vdsm-4.14.13-1.el6ev | Doc Type: | Bug Fix | ||||
| Doc Text: |
Previously, if sanlock was present on a system during upgrade from version 3.3 to 3.4, the system attempted to stop and restart sanlock. The update process would finish, but with vdsm-tool exceptions. Now, sanlock restart is requested by vdsm-tool only if there is a relevant change.
|
Story Points: | --- | ||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2014-09-04 12:59:33 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | Infra | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1123858 | ||||||
| Attachments: |
|
||||||
upgrading from 4.13.4-0.el6 to 4.14.6-0.el6, see output below: maybe the bug relates to hosted-engine utility ? does it disconnect the host from all connected SPs if connected? Federico, the reconfigure needs to run in that (3.3 to 3.4) case, means restart for sanlock might occur forcefully although we try to prevent it if not required (still not in reported version). so do you consider this case as bug if we don't restart sanlock anymore? who can reproduce and check it with hosted-engine utility? There are unfinished transactions remaining. You might consider running yum-complete-transaction first to finish them. --> Running transaction check ---> Package vdsm.x86_64 0:4.13.4-0.el6 will be updated ---> Package vdsm.x86_64 0:4.14.6-0.el6 will be an update --> Processing Dependency: vdsm-xmlrpc = 4.14.6-0.el6 for package: vdsm-4.14.6-0.el6.x86_64 --> Processing Dependency: vdsm-python-zombiereaper = 4.14.6-0.el6 for package: vdsm-4.14.6-0.el6.x86_64 --> Processing Dependency: vdsm-python = 4.14.6-0.el6 for package: vdsm-4.14.6-0.el6.x86_64 --> Running transaction check ---> Package vdsm-python.x86_64 0:4.13.4-0.el6 will be updated --> Processing Dependency: vdsm-python = 4.13.4-0.el6 for package: vdsm-cli-4.13.4-0.el6.noarch ---> Package vdsm-python.x86_64 0:4.14.6-0.el6 will be an update ---> Package vdsm-python-zombiereaper.noarch 0:4.14.6-0.el6 will be installed ---> Package vdsm-xmlrpc.noarch 0:4.13.4-0.el6 will be updated ---> Package vdsm-xmlrpc.noarch 0:4.14.6-0.el6 will be an update --> Running transaction check ---> Package vdsm-cli.noarch 0:4.13.4-0.el6 will be updated ---> Package vdsm-cli.noarch 0:4.14.6-0.el6 will be an update --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================================================================== Updating: vdsm x86_64 4.14.6-0.el6 ovirt-latest 797 k Installing for dependencies: vdsm-python-zombiereaper noarch 4.14.6-0.el6 ovirt-latest 17 k Updating for dependencies: vdsm-cli noarch 4.14.6-0.el6 ovirt-latest 68 k vdsm-python x86_64 4.14.6-0.el6 ovirt-latest 136 k vdsm-xmlrpc noarch 4.14.6-0.el6 ovirt-latest 34 k Transaction Summary ============================================================================================================================================================================================================================================== Install 1 Package(s) Upgrade 4 Package(s) Total download size: 1.0 M Is this ok [y/N]: y Downloading Packages: (1/5): vdsm-4.14.6-0.el6.x86_64.rpm | 797 kB 00:06 (2/5): vdsm-cli-4.14.6-0.el6.noarch.rpm | 68 kB 00:00 (3/5): vdsm-python-4.14.6-0.el6.x86_64.rpm | 136 kB 00:00 (4/5): vdsm-python-zombiereaper-4.14.6-0.el6.noarch.rpm | 17 kB 00:00 (5/5): vdsm-xmlrpc-4.14.6-0.el6.noarch.rpm | 34 kB 00:00 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 102 kB/s | 1.0 MB 00:10 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : vdsm-python-zombiereaper-4.14.6-0.el6.noarch 1/9 Updating : vdsm-python-4.14.6-0.el6.x86_64 2/9 Updating : vdsm-xmlrpc-4.14.6-0.el6.noarch 3/9 Updating : vdsm-cli-4.14.6-0.el6.noarch 4/9 Updating : vdsm-4.14.6-0.el6.x86_64 5/9 warning: /etc/vdsm/vdsm.conf created as /etc/vdsm/vdsm.conf.rpmnew Checking configuration status... Running configure... Done configuring modules to VDSM. Cleanup : vdsm-cli-4.13.4-0.el6.noarch 6/9 Cleanup : vdsm-4.13.4-0.el6.x86_64 7/9 Cleanup : vdsm-xmlrpc-4.13.4-0.el6.noarch 8/9 Cleanup : vdsm-python-4.13.4-0.el6.x86_64 9/9 Verifying : vdsm-cli-4.14.6-0.el6.noarch 1/9 Verifying : vdsm-4.14.6-0.el6.x86_64 2/9 Verifying : vdsm-xmlrpc-4.14.6-0.el6.noarch 3/9 Verifying : vdsm-python-zombiereaper-4.14.6-0.el6.noarch 4/9 Verifying : vdsm-python-4.14.6-0.el6.x86_64 5/9 Verifying : vdsm-4.13.4-0.el6.x86_64 6/9 Verifying : vdsm-xmlrpc-4.13.4-0.el6.noarch 7/9 Verifying : vdsm-python-4.13.4-0.el6.x86_64 8/9 Verifying : vdsm-cli-4.13.4-0.el6.noarch 9/9 Dependency Installed: vdsm-python-zombiereaper.noarch 0:4.14.6-0.el6 Updated: vdsm.x86_64 0:4.14.6-0.el6 If the host is really in maintenance (no lockspaces) sanlock must restart without any issue. I think that hosted engine even when in maintenance it is not releasing the lockspaces (bug). Anyway sanlock restart should be requested by vdsm-tool only if there was a relevant change (e.g. sanlock added to the qemu group, etc...). Created attachment 899174 [details]
vdsm logs from nott-vds1.lab.qa.tlv.redhat.com and engine log from 10.35.161.23
Checked on update from vdsm-4.13.2-0.15.el6ev.x86_64 to vdsm-4.14.7-5.el6ev.x86_64, exception still exist:
Total 11 MB/s | 1.3 MB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : vdsm-python-zombiereaper-4.14.7-5.el6ev.noarch 1/13
Updating : vdsm-python-4.14.7-5.el6ev.x86_64 2/13
Updating : vdsm-xmlrpc-4.14.7-5.el6ev.noarch 3/13
Updating : vdsm-cli-4.14.7-5.el6ev.noarch 4/13
Updating : vdsm-4.14.7-5.el6ev.x86_64 5/13
warning: /etc/vdsm/vdsm.conf created as /etc/vdsm/vdsm.conf.rpmnew
Checking configuration status...
Traceback (most recent call last):
File "/usr/bin/vdsm-tool", line 145, in <module>
sys.exit(main())
File "/usr/bin/vdsm-tool", line 142, in main
return tool_command[cmd]["command"](*args[1:])
File "/usr/lib64/python2.6/site-packages/vdsm/tool/configurator.py", line 239, in configure
service.service_stop(s)
File "/usr/lib64/python2.6/site-packages/vdsm/tool/service.py", line 370, in service_stop
return _runAlts(_srvStopAlts, srvName)
File "/usr/lib64/python2.6/site-packages/vdsm/tool/service.py", line 351, in _runAlts
"%s failed" % alt.func_name, out, err)
vdsm.tool.service.ServiceOperationError: ServiceOperationError: _serviceStop failed
Sending stop signal sanlock (2868): [ OK ]
Waiting for sanlock (2868) to stop:[FAILED]
To clarify, when I change hosted-engine --set-maintenance --mode=global, I just say to HA-agent to stop looking on engine vm, but connection with storage still exist, because if you have number of hosts in hosted-engine environment, they must know status of other host, so sanlock still exist.
The question if it some suitable way to catch and handle this exception
yes you're right. don't know how I missed that. please retest with http://gerrit.ovirt.org/#/c/29992 Verified Update from vdsm-4.13.2-0.18.el6ev.x86_64 to vdsm-4.14.13-1.el6ev.x86_64 No any exceptions and vdsm up after update. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-1152.html |
Description of problem: Appear vdsm-tool exception in time of vdsm package upgrade from 3.3 to 3.4, if in system exist sanlock(in my case it was hosted-engine that achieve sanlock) Version-Release number of selected component (if applicable): upgrade from vdsm-4.13.2-0.13.el6ev.x86_64 to vdsm-4.14.6-0.1.beta3.el6ev.x86_64 How reproducible: Always Steps to Reproduce: 1. run hosted-engine --deploy and finish process for versions 3.3 2. update repo on host to 3.4 and run hosted-engine --set-maintenance --mode 3. run yum update -y Actual results: Update process finish fine but in middle drop exception: Traceback (most recent call last): File "/usr/bin/vdsm-tool", line 145, in <module> sys.exit(main()) File "/usr/bin/vdsm-tool", line 142, in main return tool_command[cmd]["command"](*args[1:]) File "/usr/lib64/python2.6/site-packages/vdsm/tool/configurator.py", line 230, in configure service.service_stop(s) File "/usr/lib64/python2.6/site-packages/vdsm/tool/service.py", line 369, in service_stop return _runAlts(_srvStopAlts, srvName) File "/usr/lib64/python2.6/site-packages/vdsm/tool/service.py", line 350, in _runAlts "%s failed" % alt.func_name, out, err) vdsm.tool.service.ServiceOperationError: ServiceOperationError: _serviceStop failed Sending stop signal sanlock (22855): [ OK ] Waiting for sanlock (22855) to stop:[FAILED] Expected results: update process must finished without any exceptions Additional info: I add reproduce steps for my case, but I am sure it exist easiest way to create sanlock on host)