Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1339126

Summary: mom.log shows "undefined symbol ksmMergeAcrossNodes" continuously if connected to RHEV-M 3.6
Product: Red Hat Enterprise Virtualization Manager Reporter: Martin Tessun <mtessun>
Component: ovirt-engineAssignee: Martin Sivák <msivak>
Status: CLOSED ERRATA QA Contact: sefi litmanovich <slitmano>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.6.0CC: bgraveno, dfediuck, gklein, kshukla, lsurette, mavital, mgoldboi, mtessun, rbalakri, rgolan, Rhev-m-bugs, srevivo, ykaul
Target Milestone: ovirt-3.6.8Keywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
The Manager no longer sends Kernel Same-Page Merging (KSM) across nodes to Red Hat Enterprise Virtualization 3.5 hosts. The Memory Overcommit Manager (MoM) was enhanced to support KSM across nodes in Red Hat Enterprise Virtualization 3.6. However, Red Hat Enterprise Virtualization 3.5 does not have the necessary defaults for it, which means a Red Hat Enterprise Virtualization 3.5 host's MoM policy evaluation fails when the version 3.6 Manager is configured to use the host in 3.5 compatibility mode.
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-27 14:11:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: SLA RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
00-defines and 03-ksm policy from the affected RHEV 3.5 host. none

Description Martin Tessun 2016-05-24 08:29:25 UTC
Description of problem:
mom.log gets flooded with the following log-lines, in case a RHEL6 is attached to a RHEV-M 3.6 (cluster compatibility 3.5):
2016-05-24 10:23:49,273 - mom.Policy - ERROR - Policy error: undefined symbol ksmMergeAcrossNodes


Version-Release number of selected component (if applicable):
vdsm-python-zombiereaper-4.16.36-1.el6ev.noarch
vdsm-python-4.16.36-1.el6ev.noarch
vdsm-jsonrpc-4.16.36-1.el6ev.noarch
vdsm-xmlrpc-4.16.36-1.el6ev.noarch
vdsm-cli-4.16.36-1.el6ev.noarch
vdsm-4.16.36-1.el6ev.x86_64
vdsm-yajsonrpc-4.16.36-1.el6ev.noarch
mom-0.4.1-4.el6ev.noarch

rhevm-3.6.5.3-0.1.el6.noarch

How reproducible:
always

Steps to Reproduce:
1. Create a DC with 3.5 compatibility level in RHEV 3.6
2. Add a Cluster in 3.5 compatibility level
3. Attach/install a RHEL6 hypervisor and add it to the cluster
4. Add a Storage Domain.
5. Observe /var/log/vdsm/mom.log

Actual results:
The mom.log looks as follows:
2016-05-24 10:10:46,764 - mom - INFO - MOM starting
2016-05-24 10:10:47,013 - mom - INFO - hypervisor interface vdsm
2016-05-24 10:10:47,015 - mom.HostMonitor - INFO - Host Monitor starting
2016-05-24 10:10:47,197 - mom.HostMonitor - INFO - HostMonitor is ready
2016-05-24 10:10:47,221 - mom.GuestManager - INFO - Guest Manager starting
2016-05-24 10:10:47,588 - mom.Policy - INFO - Loaded policy '00-defines'
2016-05-24 10:10:47,822 - mom.Policy - INFO - Loaded policy '02-balloon'
2016-05-24 10:10:47,858 - mom.Policy - INFO - Loaded policy '03-ksm'
2016-05-24 10:10:47,933 - mom.Policy - INFO - Loaded policy '04-cputune'
2016-05-24 10:10:47,947 - mom.PolicyEngine - INFO - Policy Engine starting
2016-05-24 10:10:47,969 - mom.RPCServer - INFO - RPC Server is disabled
2016-05-24 10:10:58,000 - mom.Controllers.KSM - INFO - Updating KSM configuration: pages_to_scan:0 run:0 sleep_millisecs:0
2016-05-24 10:11:06,199 - mom.RPCServer - INFO - setNamedPolicy()
2016-05-24 10:11:06,252 - mom.Policy - INFO - Loaded policy '01-parameters'
2016-05-24 10:11:08,017 - mom.Policy - ERROR - Policy error: undefined symbol ksmMergeAcrossNodes
2016-05-24 10:11:18,031 - mom.Policy - ERROR - Policy error: undefined symbol ksmMergeAcrossNodes
2016-05-24 10:11:28,046 - mom.Policy - ERROR - Policy error: undefined symbol ksmMergeAcrossNodes
[...]

Expected results:
The lines
"2016-05-24 10:11:28,046 - mom.Policy - ERROR - Policy error: undefined symbol ksmMergeAcrossNodes"
should not appear

Additional info:

Comment 1 Martin Sivák 2016-05-25 11:58:52 UTC
Can you please attach the /etc/vdsm/mom.d/00-defines.policy and 03-ksm.policy files? Also changing mom's log level to DEBUG might be a good idea.

Comment 2 Martin Tessun 2016-05-25 13:10:27 UTC
Created attachment 1161420 [details]
00-defines and 03-ksm policy from the affected RHEV 3.5 host.

Hi Martin,

I attached the two files.

Changing the verbosity to DEBUG did not show much difference:

2016-05-25 15:09:04,762 - mom.Policy - DEBUG - Results: [0, 1, 1, 0, 1, 0.20000000000000001, 0.050000000000000003, 0.20000000000000001, 0.050000000000000003, 0.0025000000000000001, 'change_big_enough', 'shrink_guest', 'grow_guest', 0.93353991682296333, 'balloon_logic', [], 'guest_qos', [], 300, -50, 64, 1250, 10, 0.20000000000000001, 'change_npages', 784832.0, 0, None, -1, 100000, 'check_and_set_quota', 'reset_quota_and_period', []]
2016-05-25 15:09:06,912 - mom.RPCServer - INFO - setNamedPolicy()
2016-05-25 15:09:06,983 - mom.Policy - INFO - Loaded policy '01-parameters'
2016-05-25 15:09:09,054 - mom.vdsmInterface - DEBUG - VM List: []
2016-05-25 15:09:14,060 - mom.vdsmInterface - DEBUG - VM List: []
2016-05-25 15:09:14,779 - mom.Policy - ERROR - Policy error: undefined symbol ksmMergeAcrossNodes
2016-05-25 15:09:19,066 - mom.vdsmInterface - DEBUG - VM List: []
2016-05-25 15:09:24,072 - mom.vdsmInterface - DEBUG - VM List: []
2016-05-25 15:09:24,793 - mom.Policy - ERROR - Policy error: undefined symbol ksmMergeAcrossNodes
2016-05-25 15:09:29,078 - mom.vdsmInterface - DEBUG - VM List: []
2016-05-25 15:09:34,084 - mom.vdsmInterface - DEBUG - VM List: []
2016-05-25 15:09:34,812 - mom.Policy - ERROR - Policy error: undefined symbol ksmMergeAcrossNodes
2016-05-25 15:09:39,091 - mom.vdsmInterface - DEBUG - VM List: []
2016-05-25 15:09:44,097 - mom.vdsmInterface - DEBUG - VM List: []
2016-05-25 15:09:44,833 - mom.Policy - ERROR - Policy error: undefined symbol ksmMergeAcrossNodes


Cheers,
Martin

Comment 3 Martin Sivák 2016-05-25 14:48:10 UTC
Ah, I know what the issue is now. I thought it is a bad upgrade, but it actually is quite the opposite.

The engine is sending a value for a variable that is not present in the RHEL 6 MOM version. We can either fix that on the engine side or on the MOM side.. I wonder what is better:

- engine not sending the value when on 3.5 cluster compatibility level
- adding a support for that value to mom although it will be ignored


This has a simple workaround fortunately. Just add the following to 00-defines.policy

(defvar ksmMergeAcrossNodes 0)
(defvar ioTuneEnabled 0)

The other one is also related to 3.6 feature and might cause the same issue if somebody enables it.

Comment 5 Roy Golan 2016-06-15 09:29:11 UTC
just small fix to prevent 3.5 cluster from sending the ksm attributes

Comment 6 Martin Sivák 2016-06-16 13:20:58 UTC
Changing the component as this has to happen on the engine side.

Comment 8 Martin Sivák 2016-06-29 11:36:11 UTC
*** Bug 1343109 has been marked as a duplicate of this bug. ***

Comment 11 sefi litmanovich 2016-07-14 08:05:11 UTC
Verified with
ENGINE:
rhevm-3.6.8-0.1.el6.noarch
HOST:
RHEL 6.7
vdsm (3.5) - vdsm-4.16.38-1.el6ev.x86_64
mom - mom-0.4.1-4.el6ev.noarch

Verified according to steps in description, the error message wasn't issued at any point in mom.log.

Comment 13 errata-xmlrpc 2016-07-27 14:11:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1507.html