Description of problem: mom.log gets flooded with the following log-lines, in case a RHEL6 is attached to a RHEV-M 3.6 (cluster compatibility 3.5): 2016-05-24 10:23:49,273 - mom.Policy - ERROR - Policy error: undefined symbol ksmMergeAcrossNodes Version-Release number of selected component (if applicable): vdsm-python-zombiereaper-4.16.36-1.el6ev.noarch vdsm-python-4.16.36-1.el6ev.noarch vdsm-jsonrpc-4.16.36-1.el6ev.noarch vdsm-xmlrpc-4.16.36-1.el6ev.noarch vdsm-cli-4.16.36-1.el6ev.noarch vdsm-4.16.36-1.el6ev.x86_64 vdsm-yajsonrpc-4.16.36-1.el6ev.noarch mom-0.4.1-4.el6ev.noarch rhevm-3.6.5.3-0.1.el6.noarch How reproducible: always Steps to Reproduce: 1. Create a DC with 3.5 compatibility level in RHEV 3.6 2. Add a Cluster in 3.5 compatibility level 3. Attach/install a RHEL6 hypervisor and add it to the cluster 4. Add a Storage Domain. 5. Observe /var/log/vdsm/mom.log Actual results: The mom.log looks as follows: 2016-05-24 10:10:46,764 - mom - INFO - MOM starting 2016-05-24 10:10:47,013 - mom - INFO - hypervisor interface vdsm 2016-05-24 10:10:47,015 - mom.HostMonitor - INFO - Host Monitor starting 2016-05-24 10:10:47,197 - mom.HostMonitor - INFO - HostMonitor is ready 2016-05-24 10:10:47,221 - mom.GuestManager - INFO - Guest Manager starting 2016-05-24 10:10:47,588 - mom.Policy - INFO - Loaded policy '00-defines' 2016-05-24 10:10:47,822 - mom.Policy - INFO - Loaded policy '02-balloon' 2016-05-24 10:10:47,858 - mom.Policy - INFO - Loaded policy '03-ksm' 2016-05-24 10:10:47,933 - mom.Policy - INFO - Loaded policy '04-cputune' 2016-05-24 10:10:47,947 - mom.PolicyEngine - INFO - Policy Engine starting 2016-05-24 10:10:47,969 - mom.RPCServer - INFO - RPC Server is disabled 2016-05-24 10:10:58,000 - mom.Controllers.KSM - INFO - Updating KSM configuration: pages_to_scan:0 run:0 sleep_millisecs:0 2016-05-24 10:11:06,199 - mom.RPCServer - INFO - setNamedPolicy() 2016-05-24 10:11:06,252 - mom.Policy - INFO - Loaded policy '01-parameters' 2016-05-24 10:11:08,017 - mom.Policy - ERROR - Policy error: undefined symbol ksmMergeAcrossNodes 2016-05-24 10:11:18,031 - mom.Policy - ERROR - Policy error: undefined symbol ksmMergeAcrossNodes 2016-05-24 10:11:28,046 - mom.Policy - ERROR - Policy error: undefined symbol ksmMergeAcrossNodes [...] Expected results: The lines "2016-05-24 10:11:28,046 - mom.Policy - ERROR - Policy error: undefined symbol ksmMergeAcrossNodes" should not appear Additional info:
Can you please attach the /etc/vdsm/mom.d/00-defines.policy and 03-ksm.policy files? Also changing mom's log level to DEBUG might be a good idea.
Created attachment 1161420 [details] 00-defines and 03-ksm policy from the affected RHEV 3.5 host. Hi Martin, I attached the two files. Changing the verbosity to DEBUG did not show much difference: 2016-05-25 15:09:04,762 - mom.Policy - DEBUG - Results: [0, 1, 1, 0, 1, 0.20000000000000001, 0.050000000000000003, 0.20000000000000001, 0.050000000000000003, 0.0025000000000000001, 'change_big_enough', 'shrink_guest', 'grow_guest', 0.93353991682296333, 'balloon_logic', [], 'guest_qos', [], 300, -50, 64, 1250, 10, 0.20000000000000001, 'change_npages', 784832.0, 0, None, -1, 100000, 'check_and_set_quota', 'reset_quota_and_period', []] 2016-05-25 15:09:06,912 - mom.RPCServer - INFO - setNamedPolicy() 2016-05-25 15:09:06,983 - mom.Policy - INFO - Loaded policy '01-parameters' 2016-05-25 15:09:09,054 - mom.vdsmInterface - DEBUG - VM List: [] 2016-05-25 15:09:14,060 - mom.vdsmInterface - DEBUG - VM List: [] 2016-05-25 15:09:14,779 - mom.Policy - ERROR - Policy error: undefined symbol ksmMergeAcrossNodes 2016-05-25 15:09:19,066 - mom.vdsmInterface - DEBUG - VM List: [] 2016-05-25 15:09:24,072 - mom.vdsmInterface - DEBUG - VM List: [] 2016-05-25 15:09:24,793 - mom.Policy - ERROR - Policy error: undefined symbol ksmMergeAcrossNodes 2016-05-25 15:09:29,078 - mom.vdsmInterface - DEBUG - VM List: [] 2016-05-25 15:09:34,084 - mom.vdsmInterface - DEBUG - VM List: [] 2016-05-25 15:09:34,812 - mom.Policy - ERROR - Policy error: undefined symbol ksmMergeAcrossNodes 2016-05-25 15:09:39,091 - mom.vdsmInterface - DEBUG - VM List: [] 2016-05-25 15:09:44,097 - mom.vdsmInterface - DEBUG - VM List: [] 2016-05-25 15:09:44,833 - mom.Policy - ERROR - Policy error: undefined symbol ksmMergeAcrossNodes Cheers, Martin
Ah, I know what the issue is now. I thought it is a bad upgrade, but it actually is quite the opposite. The engine is sending a value for a variable that is not present in the RHEL 6 MOM version. We can either fix that on the engine side or on the MOM side.. I wonder what is better: - engine not sending the value when on 3.5 cluster compatibility level - adding a support for that value to mom although it will be ignored This has a simple workaround fortunately. Just add the following to 00-defines.policy (defvar ksmMergeAcrossNodes 0) (defvar ioTuneEnabled 0) The other one is also related to 3.6 feature and might cause the same issue if somebody enables it.
just small fix to prevent 3.5 cluster from sending the ksm attributes
Changing the component as this has to happen on the engine side.
*** Bug 1343109 has been marked as a duplicate of this bug. ***
Verified with ENGINE: rhevm-3.6.8-0.1.el6.noarch HOST: RHEL 6.7 vdsm (3.5) - vdsm-4.16.38-1.el6ev.x86_64 mom - mom-0.4.1-4.el6ev.noarch Verified according to steps in description, the error message wasn't issued at any point in mom.log.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-1507.html