Created attachment 965602 [details] logs Description of problem: The flow is Part of Rhev-Upgrade simulation: A costumer is having 3.4 DC with iSCSi and NFS domains, DC has 2X clusters 3.4 DC1 --> HC1_1 | --> brown-vdsc (3.4) --> black-vdsb (3.4) --> HC1_2 | --> alma04 (3.5) after alma04 upgrade, The costumer upgrade HC1_2 cluster policy to 3.5 as well. after cluster's upgrade, engine attempts to create and update OvdStore images on DC1, operation fails and throws NPE every $OvfUpdateIntervalInMinutes. from logs: 2014-12-07 18:58:29,610 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: java.lang.NullPointerException 2014-12-07 18:58:29,611 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) [37353cb1] Successfully removed unneeded template/vm OVFs in Data Center DC1_2 2014-12-07 18:58:29,614 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Failed to refresh VDS , vds = 7ccb3feb-8c10-4a33-b425-b0126479d487 : alma04.qa.l ab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException 2014-12-07 18:58:32,872 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-90) Failure to refresh Vds runtime info: java.lang.NullPointerException 2014-12-07 18:58:32,876 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) Failed to refresh VDS , vds = 7ccb3feb-8c10-4a33-b425-b0126479d487 : alma04.qa.l ab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException 2014-12-07 18:58:36,126 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) Failure to refresh Vds runtime info: java.lang.NullPointerException 2014-12-07 18:58:36,129 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) Failed to refresh VDS , vds = 7ccb3feb-8c10-4a33-b425-b0126479d487 : alma04.qa.l ab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException 2014-12-07 18:58:39,387 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) Failure to refresh Vds runtime info: java.lang.NullPointerException 2014-12-07 18:58:39,392 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) Failed to refresh VDS , vds = 7ccb3feb-8c10-4a33-b425-b0126479d487 : alma04.qa.l ab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException 2014-12-07 18:58:40,309 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) Failure to refresh Vds runtime info: java.lang.NullPointerException 2014-12-07 18:58:40,326 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) Failed to refresh VDS , vds = 517e6de1-28b7-4f39-8589-4c82a4509235 : brown-vdsc. qa.lab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException 2014-12-07 18:58:42,243 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) Failure to refresh Vds runtime info: java.lang.NullPointerException 2014-12-07 18:58:42,258 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) Failed to refresh VDS , vds = 133687ed-7121-4f09-9f28-62f0208af60a : black-vdsb. qa.lab.tlv.redhat.com_H, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException 2014-12-07 18:58:42,555 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) Failure to refresh Vds runtime info: java.lang.NullPointerException 2014-12-07 18:58:42,574 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) Failed to refresh VDS , vds = 7ccb3feb-8c10-4a33-b425-b0126479d487 : alma04.qa.l ab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException 2014-12-07 18:58:45,849 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: java.lang.NullPointerException 2014-12-07 18:58:45,853 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Failed to refresh VDS , vds = 7ccb3feb-8c10-4a33-b425-b0126479d487 : alma04.qa.l ab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException Version-Release number of selected component (if applicable): mixed How reproducible: 100% Expected results: engine should only try to create|update OvfStore images on 3.5 DC Additional info:
Liron, please take a look?
Sorry, meant to assign to Liron - wrong window.
The errors in the log are of VdsManager when trying to refresh the host, that's not part of the ovf update process. moving to infra.
Does it continue like that forever? Does it eventually gets fixed?
Log shows that it is part of the baloon check. Moving to sla for further examination. 2014-12-07 17:42:35,128 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) Failed to refresh VDS , vds = b2304cbd-2e0f-487d-9389-617ca306c94d : alma03.qa.lab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedBalloonCheck(VdsUpdateRunTimeInfo.java:1481) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:948) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:658) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:494) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor85.invoke(Unknown Source) [:1.7.0_71] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_71] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_71] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
the state still continues, it seems to me it will stabilize upon upgrade completion, currently still have running tests on mixed environment(3.4-3.5)
*** Bug 1171824 has been marked as a duplicate of this bug. ***
Works for me on these components: libvirt-0.10.2-46.el6_6.2.x86_64 vdsm-4.16.8.1-3.el6ev.x86_64 ovirt-hosted-engine-setup-1.2.1-8.el6ev.noarch sanlock-2.8-1.el6.x86_64 ovirt-host-deploy-1.3.0-2.el6ev.noarch ovirt-hosted-engine-ha-1.2.4-3.el6ev.noarch mom-0.4.1-4.el6ev.noarch rhevm-3.5.0-0.25.el6ev.noarch
rhev 3.5.0 was released. closing.