Bug 1171491 - [Rhev-Upgrade] NPE when refreshing a host (VdsUpdateRunTimeInfo) after upgrade
Summary: [Rhev-Upgrade] NPE when refreshing a host (VdsUpdateRunTimeInfo) after upgrade
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.5.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 3.5.0
Assignee: Doron Fediuck
QA Contact: Nikolai Sednev
URL:
Whiteboard: sla
: 1171824 (view as bug list)
Depends On:
Blocks: 1157378 rhev35rcblocker rhev35gablocker
TreeView+ depends on / blocked
 
Reported: 2014-12-07 17:21 UTC by Ori Gofen
Modified: 2016-05-26 01:49 UTC (History)
17 users (show)

Fixed In Version: org.ovirt.engine-root-3.5.0-25
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-02-17 17:14:19 UTC
oVirt Team: SLA
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
logs (6.46 MB, application/x-gzip)
2014-12-07 17:21 UTC, Ori Gofen
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 36016 0 master MERGED core: fixed NPE when vdsm returns emtpy balloon info Never
oVirt gerrit 36066 0 ovirt-engine-3.5 MERGED core: fixed NPE when vdsm returns emtpy balloon info Never

Description Ori Gofen 2014-12-07 17:21:40 UTC
Created attachment 965602 [details]
logs

Description of problem:
The flow is Part of Rhev-Upgrade simulation:
A costumer is having 3.4 DC with iSCSi and NFS domains, DC has 2X clusters

3.4 DC1
  --> HC1_1
      |
       --> brown-vdsc (3.4)
       --> black-vdsb (3.4)
  --> HC1_2
      |
       --> alma04 (3.5)

after alma04 upgrade, The costumer upgrade HC1_2 cluster policy to 3.5 as well.
after cluster's upgrade, engine attempts to create and update OvdStore images on DC1, operation fails and throws NPE every $OvfUpdateIntervalInMinutes.

from logs:

2014-12-07 18:58:29,610 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: java.lang.NullPointerException

2014-12-07 18:58:29,611 INFO  [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) [37353cb1] Successfully removed unneeded template/vm OVFs in Data Center DC1_2
2014-12-07 18:58:29,614 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Failed to refresh VDS , vds = 7ccb3feb-8c10-4a33-b425-b0126479d487 : alma04.qa.l
ab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException

2014-12-07 18:58:32,872 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-90) Failure to refresh Vds runtime info: java.lang.NullPointerException

2014-12-07 18:58:32,876 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) Failed to refresh VDS , vds = 7ccb3feb-8c10-4a33-b425-b0126479d487 : alma04.qa.l
ab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException

2014-12-07 18:58:36,126 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) Failure to refresh Vds runtime info: java.lang.NullPointerException

2014-12-07 18:58:36,129 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) Failed to refresh VDS , vds = 7ccb3feb-8c10-4a33-b425-b0126479d487 : alma04.qa.l
ab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException

2014-12-07 18:58:39,387 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) Failure to refresh Vds runtime info: java.lang.NullPointerException

2014-12-07 18:58:39,392 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) Failed to refresh VDS , vds = 7ccb3feb-8c10-4a33-b425-b0126479d487 : alma04.qa.l
ab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException

2014-12-07 18:58:40,309 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) Failure to refresh Vds runtime info: java.lang.NullPointerException

2014-12-07 18:58:40,326 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) Failed to refresh VDS , vds = 517e6de1-28b7-4f39-8589-4c82a4509235 : brown-vdsc.
qa.lab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException

2014-12-07 18:58:42,243 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) Failure to refresh Vds runtime info: java.lang.NullPointerException

2014-12-07 18:58:42,258 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) Failed to refresh VDS , vds = 133687ed-7121-4f09-9f28-62f0208af60a : black-vdsb.
qa.lab.tlv.redhat.com_H, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException

2014-12-07 18:58:42,555 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) Failure to refresh Vds runtime info: java.lang.NullPointerException

2014-12-07 18:58:42,574 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) Failed to refresh VDS , vds = 7ccb3feb-8c10-4a33-b425-b0126479d487 : alma04.qa.l
ab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException

2014-12-07 18:58:45,849 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) Failure to refresh Vds runtime info: java.lang.NullPointerException

2014-12-07 18:58:45,853 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) Failed to refresh VDS , vds = 7ccb3feb-8c10-4a33-b425-b0126479d487 : alma04.qa.l
ab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException


Version-Release number of selected component (if applicable):
mixed

How reproducible:
100%

Expected results:
engine should only try to create|update OvfStore images on 3.5 DC

Additional info:

Comment 1 Allon Mureinik 2014-12-08 11:20:43 UTC
Liron, please take a look?

Comment 2 Allon Mureinik 2014-12-08 13:08:40 UTC
Sorry, meant to assign to Liron - wrong window.

Comment 3 Liron Aravot 2014-12-08 14:39:39 UTC
The errors in the log are of VdsManager when trying to refresh the host, that's not part of the ovf update process.

moving to infra.

Comment 4 Oved Ourfali 2014-12-08 14:53:58 UTC
Does it continue like that forever?
Does it eventually gets fixed?

Comment 5 Oved Ourfali 2014-12-08 14:57:06 UTC
Log shows that it is part of the baloon check. Moving to sla for further examination.

2014-12-07 17:42:35,128 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) Failed to refresh VDS , vds = b2304cbd-2e0f-487d-9389-617ca306c94d : alma03.qa.lab.tlv.redhat.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException
	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedBalloonCheck(VdsUpdateRunTimeInfo.java:1481) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:948) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:658) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:494) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:]
	at sun.reflect.GeneratedMethodAccessor85.invoke(Unknown Source) [:1.7.0_71]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_71]
	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_71]
	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:]
	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]

Comment 6 Ori Gofen 2014-12-10 07:30:35 UTC
the state still continues, it seems to me it will stabilize upon upgrade completion, currently still have running tests on mixed environment(3.4-3.5)

Comment 7 Doron Fediuck 2014-12-15 13:59:11 UTC
*** Bug 1171824 has been marked as a duplicate of this bug. ***

Comment 8 Nikolai Sednev 2014-12-16 09:04:43 UTC
Works for me on these components: 
libvirt-0.10.2-46.el6_6.2.x86_64
vdsm-4.16.8.1-3.el6ev.x86_64
ovirt-hosted-engine-setup-1.2.1-8.el6ev.noarch
sanlock-2.8-1.el6.x86_64
ovirt-host-deploy-1.3.0-2.el6ev.noarch
ovirt-hosted-engine-ha-1.2.4-3.el6ev.noarch
mom-0.4.1-4.el6ev.noarch
rhevm-3.5.0-0.25.el6ev.noarch

Comment 10 Eyal Edri 2015-02-17 17:14:19 UTC
rhev 3.5.0 was released. closing.


Note You need to log in before you can comment on or make changes to this bug.