Bug 920671 - [rhevh upgrade] Reporting a 'Failed to upgrade' to engine, while it really succeeded
Summary: [rhevh upgrade] Reporting a 'Failed to upgrade' to engine, while it really su...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 3.2.0
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 3.2.0
Assignee: Douglas Schilling Landgraf
QA Contact: Artyom
URL:
Whiteboard: infra
: 928589 928699 978235 1434728 (view as bug list)
Depends On:
Blocks: 1040163
TreeView+ depends on / blocked
 
Reported: 2013-03-12 14:09 UTC by Tareq Alayan
Modified: 2022-07-09 05:58 UTC (History)
28 users (show)

Fixed In Version: vdsm-4.10.2-12.0
Doc Type: Release Note
Doc Text:
After a Red Hat Enterprise Virtualization Hypervisor is attached to the Red Hat Enterprise Virtualization Manager and then successfully upgraded, it may erroneously appear in the administration portal with the status of Install Failed. Click on the Activate button, and the hypervisor will change to an Up status and be ready for use.
Clone Of:
: 1040163 (view as bug list)
Environment:
Last Closed: 2013-05-23 12:26:02 UTC
oVirt Team: Infra
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
engine.log.3.1 (6.01 KB, application/x-gzip)
2013-03-12 15:46 UTC, Tareq Alayan
no flags Details
engine.log.3.2 (32.64 KB, application/x-gzip)
2013-03-12 15:46 UTC, Tareq Alayan
no flags Details
engine.log.3.2.sf11 (7.34 KB, application/x-gzip)
2013-03-22 09:01 UTC, Tareq Alayan
no flags Details
engine.log.sf14 (139.04 KB, text/plain)
2013-04-26 09:52 UTC, cshao
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHV-47078 0 None None None 2022-07-09 05:58:40 UTC
Red Hat Knowledge Base (Solution) 380313 0 None None None Never
Red Hat Product Errata RHSA-2013:0886 0 normal SHIPPED_LIVE Moderate: rhev 3.2 - vdsm security and bug fix update 2013-06-11 00:25:02 UTC
oVirt gerrit 12990 0 None None None Never
oVirt gerrit 15023 0 None None None Never

Description Tareq Alayan 2013-03-12 14:09:37 UTC
Description of problem:
Error in reporting upgrade successful to engine when upgrading rhevh 

Version-Release number of selected component (if applicable):
sf10

How reproducible:
always


Steps to Reproduce:
1. have rhevh installed and up (version 20130227)
2. have iso for a newer rhevh 
3. upgrade host from GUI
  
Actual results:
On engine UI says that installation failed

Expected results:
upgraded successfully and state is UP 


Additional info:
snippet from engine log:
Installation of Host. Received illegal XML from Host. Message: <BSTRAP component='RHEL_INSTALL'
status='OK'message='vdsm daemon stopped for upgrade process!'

looks like a type in the xml

Comment 1 Tareq Alayan 2013-03-12 15:45:04 UTC
same scenario with 3.1 engine si27.4: 
The host upgraded successfully and becomes up, however it takes ~6 minutes.
There are some error messages in the engine.log worth take a look at them [1]:
==============================================================================
2013-03-12 05:22:00,109 ERROR [org.ovirt.engine.core.bll.InstallerMessages] (NioProcessor-3) Installation of Host. Received illegal XML from Host. Message: org.ovirt.engine.core.compat.CompatException: Element type "BSTRAP" must be followed by either attribute specifications, ">" or "/>"., Exception: {2}
...
2013-03-12 05:23:41,864 ERROR [org.ovirt.engine.core.bll.RegisterVdsQuery] (ajp-/127.0.0.1:8702-7) Query execution failed due to invalid inputs. VDS_STATUS_NOT_VALID_FOR_UPDATE



attached:
[1] engine.log.3.1
[2] engine.log.3.2

Comment 2 Tareq Alayan 2013-03-12 15:46:08 UTC
Created attachment 709031 [details]
engine.log.3.1

Comment 3 Tareq Alayan 2013-03-12 15:46:56 UTC
Created attachment 709032 [details]
engine.log.3.2

Comment 6 Tareq Alayan 2013-03-22 09:01:35 UTC
Created attachment 714385 [details]
engine.log.3.2.sf11

Comment 7 Tareq Alayan 2013-03-22 09:03:09 UTC
Upgrade fails on 3.2 sf11.
attached engine.log 3.2 sf11

Comment 8 Douglas Schilling Landgraf 2013-03-22 23:19:09 UTC
Hello Tareq,

Indeed, from logs I can see:
2013-03-22 10:53:36,176 INFO  [org.ovirt.engine.core.bll.OVirtNodeUpgrade] (OVirtNodeUpgrade) update from host 10.35.102.79: <BSTRAP component='RHEL_INSTALL' status='OK'message='vdsm daemon stopped for upgrade process!'/>
2013-03-22 10:53:36,187 ERROR [org.ovirt.engine.core.bll.InstallerMessages] (OVirtNodeUpgrade) Installation of Host. Received illegal XML from Host. Message: <BSTRAP component='RHEL_INSTALL' status='OK'message='vdsm daemon stopped for upgrade process!'/>: org.ovirt.engine.core.compat.CompatException: Element type "BSTRAP" must be followed by either attribute specifications, ">" or "/>".

However, which RHEV-H have you used in your tests? 

The last one which I can see from the rhev-devel available is:
RHEVH-ADV 3.2 Beta 1 (6.4-20130321.0) which contain vdsm-4.10.2-11.0.el6ev.
Unfortunately, this image doesn't contain the fix for this bug.

vdsm-4.10.2-12.0 [1] should resolve this issue.
I am moving back to modified state, fell free to move it back to assigned in case this version still doesn't resolve the problem.

[1] https://brewweb.devel.redhat.com/buildinfo?buildID=263212

Thanks
Douglas

Comment 9 Tareq Alayan 2013-03-24 09:46:46 UTC
I used rhev-hypervisor6-6.4-20130318.1.el6_4
The engine version was 3.2 sf11.

In case the engine version is 3.1.z, the upgrade succeed and the state is up. [see comment 1]

Comment 10 Douglas Schilling Landgraf 2013-03-25 18:14:38 UTC
Thanks Tareq, indeed  rhev-hypervisor6-6.4-20130318.1.el6_4 contains vdsm-4.10.2-1.8.el6ev which doesn't have the fix.

Comment 11 Pavel Stehlik 2013-03-28 09:44:10 UTC
*** Bug 928699 has been marked as a duplicate of this bug. ***

Comment 12 Alon Bar-Lev 2013-03-29 19:15:03 UTC
*** Bug 928589 has been marked as a duplicate of this bug. ***

Comment 13 Cheryn Tan 2013-04-03 07:01:44 UTC
This bug is currently attached to errata RHBA-2012:14332. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag.

Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information:

* Cause: What actions or circumstances cause this bug to present.

* Consequence: What happens when the bug presents.

* Fix: What was done to fix the bug.

* Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')

Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug.

For further details on the Cause, Consequence, Fix, Result format please refer to:

https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes

Thanks in advance.

Comment 19 cshao 2013-04-26 09:52:20 UTC
Created attachment 740330 [details]
engine.log.sf14

Comment 23 Artyom 2013-05-05 08:01:29 UTC
After return on steps that described in comment 6 of bug https://bugzilla.redhat.com/show_bug.cgi?id=950271, was successes to do upgrade from version rhev-hypervisor-advanced-6.4-20130328.0.el6ev to rhev-hypervisor-advanced-6.4-20130425.0.el6ev(vdsm-4.10.2-13.0.el6ev -> vdsm-4.10.2-16.0.el6ev) without any error message. Version of rhevm sf14.

Comment 25 Ying Cui 2013-05-20 11:12:10 UTC
rhev-hypervisor-6.4-20130501.0.auto468 with vdsm-4.10.2-19.0.el6ev

Comment 27 wanghui 2013-05-21 02:42:15 UTC
Hi Douglas,

As you comment#26, is this means the scenario that upgrade rhevh 6.4.z with vdsm3.1 to the rhevh 6.4.z with vdsm3.2 which newer than vdsm-4.10.2-12.0 will not be supported on RHEV-M SF 3.1 cluster? 

rhev-hypervisor-6.4-20130318.1.el6_4 with vdsm-4.10.2-1.8.el6ev.x86_64
rhev-hypervisor-6.4-20130501.0.auto468/rhev-hypervisor6-6.4-20130501.0.auto635.el6_4 with vdsm-4.10.2-19.0.el6ev
 	

RHEV-H 6.4.z with vdsm3.1 upgrade to RHEV-H 6.4.z with vdsm3.1 latest on RHEV-M SI 3.1 cluster. PASS
RHEV-H 6.4.z with vdsm3.1 upgrade to RHEV-H 6.4.z with vdsm3.2 latest on RHEV-M SF 3.1 cluster. FAILED

Comment 36 Alon Bar-Lev 2013-05-23 12:13:52 UTC
host-deploy: ignore malformed messages received from host

CURRENT STATUS

Legacy vdsm-upgrade reports its status via semi-XML format. Unfortunately
there was a bug in vdsm-upgrade in which the XML that was sent had been
malformed.

As a result the upgrade of the ovirt-node succeeded, but host was marked
as installed failed.

WORKAROUND

After upgrade from ovirt-node with this bug, move host to maintain once and
then activate.

NEW IMPLEMENTATION

Do not fail installation if malformed data is received, issue a warning.

IMPLICATIONS

Upgrade will not fail if a severe error occurred at host side, for
example python exception.

SUGGESTION

Not to apply this patch, but provide a known issues paragraph in
documentation instead.

Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=920671
Change-Id: I05262d92dfbb76de75a66a32c83998ad0361c62d
Signed-off-by: Alon Bar-Lev <alonbl>

Comment 38 Alon Bar-Lev 2013-06-26 11:12:30 UTC
*** Bug 978235 has been marked as a duplicate of this bug. ***

Comment 39 Barak 2013-07-01 14:18:19 UTC
*** Bug 974368 has been marked as a duplicate of this bug. ***

Comment 40 Lee Yarwood 2013-07-31 11:06:01 UTC
*** Bug 978235 has been marked as a duplicate of this bug. ***

Comment 41 Alon Bar-Lev 2013-10-17 07:53:44 UTC
*** Bug 1019794 has been marked as a duplicate of this bug. ***

Comment 42 Douglas Schilling Landgraf 2017-03-22 21:14:08 UTC
*** Bug 1434728 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.