Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1151488

Summary: [ppc] Host is installed with VDSM version (<UNKNOWN>) and cannot join cluster ppc64...
Product: Red Hat Enterprise Virtualization Manager Reporter: Jiri Belka <jbelka>
Component: ovirt-engineAssignee: Yaniv Bronhaim <ybronhei>
Status: CLOSED CURRENTRELEASE QA Contact: Jiri Belka <jbelka>
Severity: high Docs Contact:
Priority: urgent    
Version: 3.5.0CC: bkorren, danken, ecohen, gklein, iheim, jbelka, lpeer, lsurette, michal.skrivanek, oourfali, pstehlik, rbalakri, Rhev-m-bugs, sherold, ybronhei, yeylon
Target Milestone: ---Keywords: Reopened, ZStream
Target Release: 3.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: infra
Fixed In Version: org.ovirt.engine-root-3.5.0-18 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1152448 (view as bug list) Environment:
Last Closed: 2015-02-17 17:17:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1122979, 1152448    
Attachments:
Description Flags
engine.log, vdsm.log none

Description Jiri Belka 2014-10-10 14:47:07 UTC
Created attachment 945662 [details]
engine.log, vdsm.log

Description of problem:

<< 3.5 engine part of the issue - <UNKNOWN> in event, cannot add into rhevm 3.5 , engine checks not only for 'supportedENGINEs but also for packages2 list for vdsm >>

RHEV for IBM POWER release 3.4 build 35 service (pkvm2_1) can't be added to engine,...

Host ibm-p8-rhevm-hv-01.lab.bos.redhat.com is installed with VDSM version (<UNKNOWN>) and cannot join cluster ppc64 which is compatible with VDSM versions [4.13, 4.14, 4.9, 4.11, 4.15, 4.12, 4.10].

...
2014-10-10 16:16:06,088 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [5139c54f] Correlation ID: 5139c54f, Job ID: 7827e052-c40b-44e2-9d50-e7939b52d326, Call Stack: null, Custom Event ID: -1, Message: Host ibm-p8-rhevm-hv-01.lab.bos.redhat.com is installed with VDSM version (<UNKNOWN>) and cannot join cluster ppc64 which is compatible with VDSM versions [4.13, 4.14, 4.9, 4.11, 4.15, 4.12, 4.10].

seems to be related to getos()...

...
    if getos() in (OSName.RHEVH, OSName.OVIRT, OSName.FEDORA, OSName.RHEL):
        KEY_PACKAGES = {'qemu-kvm': ('qemu-kvm', 'qemu-kvm-rhev'),
                        'qemu-img': ('qemu-img', 'qemu-img-rhev'),
                        'vdsm': ('vdsm',),
                        'spice-server': ('spice-server',),
                        'libvirt': ('libvirt', 'libvirt-daemon-kvm'),
                        'mom': ('mom',),
                        }


...
        operatingSystem = {'name': 'PowerKVM', 'release': '3.4.2.35.0.pkvm2_1', 'version': '19_0'}
        packages2 = {'kernel': {'buildtime': 1412189087.0,
                                'release': '1709.pkvm2_1.12.ppc64',
                                'version': '3.10.23'}}


workaround:

--- /tmp/caps.py.orig   2014-10-10 10:27:47.976434065 -0400
+++ /usr/share/vdsm/caps.py     2014-10-10 10:26:32.235548944 -0400
@@ -443,7 +443,7 @@ def _getKeyPackages():
 
     pkgs = {'kernel': kernelDict()}
 
-    if getos() in (OSName.RHEVH, OSName.OVIRT, OSName.FEDORA, OSName.RHEL):
+    if getos() in (OSName.RHEVH, OSName.OVIRT, OSName.FEDORA, OSName.RHEL, OSName.POWERKVM):
         KEY_PACKAGES = {'qemu-kvm': ('qemu-kvm', 'qemu-kvm-rhev'),
                         'qemu-img': ('qemu-img', 'qemu-img-rhev'),
                         'vdsm': ('vdsm',),

with patch:

...
        packages2 = {'kernel': {'buildtime': 1412189087.0,
                                'release': '1709.pkvm2_1.12.ppc64',
                                'version': '3.10.23'},
                     'libvirt': {'buildtime': 1412717569,
                                 'release': '1.pkvm2_1.17.9',
                                 'version': '1.1.3'},
                     'mom': {'buildtime': 1405446678,
                             'release': '1.pkvm2_1.1',
                             'version': '0.4.1'},
                     'qemu-img': {'buildtime': 1412195719,
                                  'release': '2.pkvm2_1.17.9',
                                  'version': '1.6.0'},
                     'qemu-kvm': {'buildtime': 1412195719,
                                  'release': '2.pkvm2_1.17.9',
                                  'version': '1.6.0'},
                     'vdsm': {'buildtime': 1412188587,
                              'release': '1.pkvm2_1',
                              'version': '4.14.17'}}
        reservedMem = '321'
        rngSources = ['hwrng', 'random']
        software_revision = '1'
        software_version = '4.14'
        supportedENGINEs = ['3.0', '3.1', '3.2', '3.3', '3.4']
        supportedProtocols = ['2.2', '2.3']
        supportedRHEVMs = ['3.0']
        uuid = '2116A5A'
        version_name = 'Snow Man'
        vlans = {}
        vmTypes = ['kvm']


Version-Release number of selected component (if applicable):
vdsm-4.14.17-1.pkvm2_1.ppc64

How reproducible:
100%

Steps to Reproduce:
1. add _new_ powerkvm for 3.4 into 3.5 (3.4 clstr obviously)
2. 
3.

Actual results:
fail

Expected results:
should work

Additional info:
mskrivanek@ said it's two issues: vdsm reporting and engine checks

Comment 3 Michal Skrivanek 2014-10-14 08:20:22 UTC
Yaniv/Oved, there needs to be a proper engine side fix as well. In 3.5 (in contrast to 3.4.z) the engine requires the packages to be reported for some reason. This is probably a regression in engine
We want the vdsm fix for sure, but the engine side needs to be addressed too

Comment 4 Oved Ourfali 2014-10-14 08:54:01 UTC
(In reply to Michal Skrivanek from comment #3)
> Yaniv/Oved, there needs to be a proper engine side fix as well. In 3.5 (in
> contrast to 3.4.z) the engine requires the packages to be reported for some
> reason. This is probably a regression in engine
> We want the vdsm fix for sure, but the engine side needs to be addressed too

As long as there are package version the engine should know to cope with that.
What additional fix you had in mind?

Comment 5 Michal Skrivanek 2014-10-14 09:19:35 UTC
3.4 engine works even when the pkgs are not reported. "packages2" should not be needed for joining the cluster. The vdsm version is reported in other field which should be used.
So it is a 3.5 regression in engine I guess; or at least a 3.5-only problem in engine.

Comment 6 Yaniv Bronhaim 2014-10-14 09:46:41 UTC
 please verify that with the attached patch and see that engine adds the host properly. afaiu if engine didn't fail the addHost to cluster in 3.4 before, its due to a bug that was fixed regarding the version validation between 3.4 to 3.5. with unknown version engine should fail it

Comment 8 Jiri Belka 2014-10-15 09:59:34 UTC
Yes it works with 3.4.3 without problem (vdsm-4.14.17-1.pkvm2_1.ppc64).

Comment 9 Dan Kenigsberg 2014-10-15 10:51:52 UTC
The vdsm-side patch would get to 3.4.4. But it seems that there has been an unintentional behavioral change in Engine. Please consider fixing it in 3.5.1.

Comment 10 Michal Skrivanek 2014-10-21 08:19:28 UTC
*** Bug 1151489 has been marked as a duplicate of this bug. ***

Comment 11 Michal Skrivanek 2014-10-21 08:20:48 UTC
please also read bug 1151489 for more details
we need to fix engine behavior in 3.5 to look at proper variables. 
Oved, can you please prioritize?

Comment 12 Yaniv Bronhaim 2014-10-21 08:26:51 UTC
We need to work with ppc host for that, and sometimes it delays the work a bit. I move that to 3.5.1 as it requires verification and debugging to understand the change in HandleVdsVersionCommand or the version parsing

Comment 13 Yaniv Bronhaim 2014-10-26 13:31:57 UTC
Starting to verify this issue and I need to clarify the answer:

Revisiting this area again (HandleVdsVersionCommand) I see again that firstly engine checks isEngineSupportedByVdsm by checking the supported engine value from getCaps (example from vdsm 3.5: supportedENGINEs = ['3.0', '3.1', '3.2', '3.3', '3.4', '3.5']). this returned normally regarding the ppc bug which lead to unreported rpm versions. 

In the case where engine supported by vdsm - like vdsm and engine 3.5, all works and you won't see the UNKNOWN eventhough vdsm version is not reported properly (you see it in the UI - VDSM Version: [N/A] in host's tab).

But if the supportedByVdsm is false, and vdsm version is not reported, such as this case, you will see the report:
Host a1a1 is installed with VDSM version (<UNKNOWN>) and cannot join cluster Default which is compatible with VDSM versions [4.13, 4.14, 4.9, 4.11, 4.15, 4.12, 4.10].

saying that, not sure what is your case. but the code is similar both in 3.5 and 3.4, it just depends on the versions you run

I intend to close it as NOTABUG.

Comment 14 Oved Ourfali 2014-10-27 05:40:00 UTC
(In reply to Yaniv Bronhaim from comment #13)
> Starting to verify this issue and I need to clarify the answer:
> 
> Revisiting this area again (HandleVdsVersionCommand) I see again that
> firstly engine checks isEngineSupportedByVdsm by checking the supported
> engine value from getCaps (example from vdsm 3.5: supportedENGINEs = ['3.0',
> '3.1', '3.2', '3.3', '3.4', '3.5']). this returned normally regarding the
> ppc bug which lead to unreported rpm versions. 
> 
> In the case where engine supported by vdsm - like vdsm and engine 3.5, all
> works and you won't see the UNKNOWN eventhough vdsm version is not reported
> properly (you see it in the UI - VDSM Version: [N/A] in host's tab).
> 
> But if the supportedByVdsm is false, and vdsm version is not reported, such
> as this case, you will see the report:
> Host a1a1 is installed with VDSM version (<UNKNOWN>) and cannot join cluster
> Default which is compatible with VDSM versions [4.13, 4.14, 4.9, 4.11, 4.15,
> 4.12, 4.10].
> 
> saying that, not sure what is your case. but the code is similar both in 3.5
> and 3.4, it just depends on the versions you run
> 
> I intend to close it as NOTABUG.

The only claim by Michal was that without applying your fix it worked well in 3.4, but not in 3.5.
I also remember that the code is the same, and I think it probably worked well in a bit different use-case. We've been doing this validation a long time now, and that's the correct one.

Closing this one. Michal - if you notice anything wrong with the logic/behavior, please reopen.

Comment 15 Jiri Belka 2014-10-27 08:53:36 UTC
I suppose the statement in #13 was never tested.

Reproduces with vt7 (3.5.0-0.17.beta.el6e) and vdsm-4.14.17-10.gitbb25d3b.el7.ppc64

2014-Oct-27, 09:49
Host ibm-p8-rhevm-03.rhts.eng.bos.redhat.com is installed with VDSM version (<UNKNOWN>) and cannot join cluster ppc64 which is compatible with VDSM versions [4.13, 4.14, 4.9, 4.11, 4.15, 4.12, 4.10].

Yes, it is _3.5_ !

According to some discussion it's strange a BZ can be switched from ASSIGNED to CLOSED/WORKSFORME.

Comment 16 Oved Ourfali 2014-10-27 09:07:23 UTC
We need to verify that the following works:
1. Work with engine 3.5
2. Work with the most updated PPC VDSM version
3. Try to add the host to the engine
4. Make sure it works

If it works well, we can close this one.
If not, we need to apply the fix to make it work.

Jiri - Yaniv can verify that if he gets access to the host.
Or you help us verify that.

The bug was closed due to understanding it works with latest 3.5 release + latest 3.4 PPC vdsm release. If it doesn't, then we have an issue. If it does, it can be either moved to MODIFIED for proper verification, or closed.

Comment 17 Jiri Belka 2014-10-27 09:41:59 UTC
(In reply to Oved Ourfali from comment #16)
> We need to verify that the following works:
> 1. Work with engine 3.5

see above

> 2. Work with the most updated PPC VDSM version

NAME="RHEV for IBM POWER"
VERSION="3.4"
ID="rhev for ibm power"
VERSION_ID="3.4"
BUILD_ID="36.0-service"
PRETTY_NAME="RHEV for IBM POWER 3.4"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:ibm:service:pkvm2_1"

> 3. Try to add the host to the engine
> 4. Make sure it works

no it does not

> If it works well, we can close this one.
> If not, we need to apply the fix to make it work.
> 
> Jiri - Yaniv can verify that if he gets access to the host.
> Or you help us verify that.
> 
> The bug was closed due to understanding it works with latest 3.5 release +
> latest 3.4 PPC vdsm release. If it doesn't, then we have an issue. If it
> does, it can be either moved to MODIFIED for proper verification, or closed.

"understanding it works" - then it should be CLOSEDCURRENTRELEASE, no?

just to clarify (again):

- vdsm is rhev for power (3.4)
- engine is d/s 3.5 (it is *obvious* the cluster level is 3.4)

Comment 18 Oved Ourfali 2014-10-27 13:55:56 UTC
The reason it works on 3.4 and not 3.5, is because the engine is 3.5, and vdsm is 3.4, so the supportedEngines entry in VDSM doesn't contain 3.5, thus we check for the package version. In 3.4 it isn't checked because 3.4 is a supported engine by VDSM.

For verification reasons, I'm moving this one to MODIFIED, to be verified once 3.4.4 is released.

Comment 19 Oved Ourfali 2014-10-29 17:44:22 UTC
Note that it can be verified only once 3.4.4 is out.

Comment 20 Michal Skrivanek 2014-11-04 08:52:18 UTC
(In reply to Oved Ourfali from comment #18)
> The reason it works on 3.4 and not 3.5, is because the engine is 3.5, and
> vdsm is 3.4, so the supportedEngines entry in VDSM doesn't contain 3.5, thus
> we check for the package version. In 3.4 it isn't checked because 3.4 is a
> supported engine by VDSM.

aah, now it makes sense. Thanks for clarification!

> For verification reasons, I'm moving this one to MODIFIED, to be verified
> once 3.4.4 is released.

yes. it's *not* in RHEV for Power build 36, that is based on 3.4.3

Comment 21 Jiri Belka 2014-11-04 10:37:36 UTC
> yes. it's *not* in RHEV for Power build 36, that is based on 3.4.3

how can we understand the relation between RHEV for Power builds and vdsm version (without comparing rpm packages inside power build)?

Comment 22 Yaniv Bronhaim 2014-11-05 17:53:13 UTC
the rpm versions are the same

Comment 23 Jiri Belka 2014-11-07 10:08:45 UTC
ok, vt9 with vdsm 4.14.18-1 (https://bugzilla.redhat.com/show_bug.cgi?id=1152448#c4)

Comment 24 Eyal Edri 2015-02-17 17:17:24 UTC
rhev 3.5.0 was released. closing.