Bug 1250919 - [host-deploy] when updating multiple packages only the latest is considered in cache timestamp
Summary: [host-deploy] when updating multiple packages only the latest is considered i...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.5.3
Hardware: All
OS: Linux
high
high
Target Milestone: ovirt-3.6.0-rc3
: 3.6.0
Assignee: Yaniv Kaul
QA Contact: Pavol Brilla
URL:
Whiteboard:
Depends On: 1252364
Blocks: 1261411
TreeView+ depends on / blocked
 
Reported: 2015-08-06 10:02 UTC by Julio Entrena Perez
Modified: 2019-09-12 08:45 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1261339 1261353 1261359 1261411 (view as bug list)
Environment:
Last Closed: 2016-03-09 21:10:58 UTC
oVirt Team: Infra
mgoldboi: Triaged+


Attachments (Terms of Use)
TS.java (730 bytes, text/plain)
2015-08-10 14:01 UTC, Alon Bar-Lev
no flags Details
TS.java (780 bytes, text/plain)
2015-08-10 14:27 UTC, Alon Bar-Lev
no flags Details
Output of TS.java (24.98 KB, text/plain)
2015-08-11 08:37 UTC, Julio Entrena Perez
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:0376 normal SHIPPED_LIVE Red Hat Enterprise Virtualization Manager 3.6.0 2016-03-10 01:20:52 UTC
Red Hat Bugzilla 1129815 None None None Never
Red Hat Knowledge Base (Solution) 2090071 None None None Never

Internal Links: 1129815

Description Julio Entrena Perez 2015-08-06 10:02:36 UTC
Description of problem:
After upgrading RHEV-M from 3.4.5 to 3.5.3, attempting to add hosts to RHEV-M fails.

Version-Release number of selected component (if applicable):
rhevm-backend-3.5.3.1-1.4.el6ev

How reproducible:
Unknown.

Steps to Reproduce:
1. Upgrade RHEV-M.
2. Try to add a new host.
3.

Actual results:
Host fails to be added:

  2015-08-05 16:33:22,561 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-17) [2c3d77b7] START, AddVdsVDSCommand(HostName = hostname, HostId = b5e65b80-a85c-47c0-8394-ffd6157c0b4d), log id: 6b8b3f41
  2015-08-05 16:33:22,561 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-17) [2c3d77b7] AddVds - entered , starting logic to add VDS b5e65b80-a85c-47c0-8394-ffd6157c0b4d
  [...]
  2015-08-05 16:33:29,235 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 74afa17c, Call Stack: null, Custom Event ID: -1, Message: Installing Host hostname. Logs at host located at: '/tmp/ovirt-host-deploy-20150805163326-pf6cid.log'.
  2015-08-05 16:33:29,279 INFO  [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host 172.20.46.118 reports unique id 30313436-3631-5A43-4A34-31323
  0323138
  2015-08-05 16:33:29,300 INFO  [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 30313436-3631-5A43-4A34-313230323138 to Host 172.20.46.118
  2015-08-05 16:33:29,394 ERROR [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Error during deploy dialog: java.lang.NullPointerException
          at org.ovirt.engine.core.bll.VdsDeploy$37.call(VdsDeploy.java:580) [bll.jar:]
          at org.ovirt.engine.core.bll.VdsDeploy$37.call(VdsDeploy.java:579) [bll.jar:]
          at org.ovirt.engine.core.bll.VdsDeploy._nextCustomizationEntry(VdsDeploy.java:668) [bll.jar:]
          at org.ovirt.engine.core.bll.VdsDeploy._threadMain(VdsDeploy.java:873) [bll.jar:]
          at org.ovirt.engine.core.bll.VdsDeploy.access$2000(VdsDeploy.java:83) [bll.jar:]
          at org.ovirt.engine.core.bll.VdsDeploy$51.run(VdsDeploy.java:969) [bll.jar:]
          at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
  
  2015-08-05 16:33:29,408 ERROR [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-7-thread-33) [74afa17c] Error during host 172.20.46.118 install: java.lang.NullPointerException
          at org.ovirt.engine.core.bll.VdsDeploy$37.call(VdsDeploy.java:580) [bll.jar:]
          at org.ovirt.engine.core.bll.VdsDeploy$37.call(VdsDeploy.java:579) [bll.jar:]
          at org.ovirt.engine.core.bll.VdsDeploy._nextCustomizationEntry(VdsDeploy.java:668) [bll.jar:]
          at org.ovirt.engine.core.bll.VdsDeploy._threadMain(VdsDeploy.java:873) [bll.jar:]
          at org.ovirt.engine.core.bll.VdsDeploy.access$2000(VdsDeploy.java:83) [bll.jar:]
          at org.ovirt.engine.core.bll.VdsDeploy$51.run(VdsDeploy.java:969) [bll.jar:]
          at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
  
  2015-08-05 16:33:29,413 ERROR [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-7-thread-33) [74afa17c] Installation 172.20.46.118: null
  2015-08-05 16:33:29,453 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-33) [74afa17c] Correlation ID: 74afa17c, Call Stack: null, Custom Event ID: -1, Message: Failed to install Host hostname. <UNKNOWN>.
  2015-08-05 16:33:29,454 ERROR [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-7-thread-33) [74afa17c] Error during host 172.20.46.118 install, prefering first exception: java.lang.NullPointerException
          at org.ovirt.engine.core.bll.VdsDeploy$37.call(VdsDeploy.java:580) [bll.jar:]
          at org.ovirt.engine.core.bll.VdsDeploy$37.call(VdsDeploy.java:579) [bll.jar:]
          at org.ovirt.engine.core.bll.VdsDeploy._nextCustomizationEntry(VdsDeploy.java:668) [bll.jar:]
          at org.ovirt.engine.core.bll.VdsDeploy._threadMain(VdsDeploy.java:873) [bll.jar:]
          at org.ovirt.engine.core.bll.VdsDeploy.access$2000(VdsDeploy.java:83) [bll.jar:]
          at org.ovirt.engine.core.bll.VdsDeploy$51.run(VdsDeploy.java:969) [bll.jar:]
          at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
  
  2015-08-05 16:33:29,454 ERROR [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-33) [74afa17c] Host installation failed for host b5e65b80-a85c-47c0-8394-ffd6157c0b4d, hostname.: java.lang.NullPointerException
          at org.ovirt.engine.core.bll.VdsDeploy$37.call(VdsDeploy.java:580) [bll.jar:]
          at org.ovirt.engine.core.bll.VdsDeploy$37.call(VdsDeploy.java:579) [bll.jar:]
          at org.ovirt.engine.core.bll.VdsDeploy._nextCustomizationEntry(VdsDeploy.java:668) [bll.jar:]
          at org.ovirt.engine.core.bll.VdsDeploy._threadMain(VdsDeploy.java:873) [bll.jar:]
          at org.ovirt.engine.core.bll.VdsDeploy.access$2000(VdsDeploy.java:83) [bll.jar:]
          at org.ovirt.engine.core.bll.VdsDeploy$51.run(VdsDeploy.java:969) [bll.jar:]
          at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
  2015-08-05 16:33:29,482 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-33) [74afa17c] START, SetVdsStatusVDSCommand(HostName = hostname, HostId = b5e65b80-a85c-47c0-8394-ffd6157c0b4d, status=InstallFailed, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 70909605
  2015-08-05 16:33:29,501 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-33) [74afa17c] FINISH, SetVdsStatusVDSCommand, log id: 70909605
  2015-08-05 16:33:29,522 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-33) [74afa17c] Correlation ID: 74afa17c, Job ID: fb04dfb2-9537-4691-bca0-13c35f0f5f79, Call Stack: null, Custom Event ID: -1, Message: Host hostname installation failed. Please refer to /var/log/ovirt-engine/engine.log and log logs under /var/log/ovirt-engine/host-deploy/ for further details..

Expected results:
Host is added successfully.

Additional info:
After manually deleting file /var/cache/ovirt-engine/ovirt-host-deploy.tar from RHEV-M, host was added successfully.

Comment 3 Alon Bar-Lev 2015-08-06 12:21:19 UTC
please provide:

1. exact engine version
2. exact ovirt-host-deploy version
3. /var/cache/ovirt-engine/ovirt-host-deploy.tar - I hope you saved it, if not we will not be able to understand what happened in this case as well.
4. tar -czf /tmp/deploy.tar.gz /usr/share/ovirt-host-deploy

thanks!

Comment 4 Alon Bar-Lev 2015-08-06 12:25:17 UTC
(In reply to Alon Bar-Lev from comment #3)
> please provide:
> 
> 1. exact engine version
> 2. exact ovirt-host-deploy version
> 3. /var/cache/ovirt-engine/ovirt-host-deploy.tar - I hope you saved it, if
> not we will not be able to understand what happened in this case as well.

need its timestamp (the original one).

> 4. tar -czf /tmp/deploy.tar.gz /usr/share/ovirt-host-deploy
> 
> thanks!

Comment 5 Julio Entrena Perez 2015-08-06 12:27:18 UTC
(In reply to Alon Bar-Lev from comment #3)
> please provide:
> 
> 1. exact engine version

$ grep rhevm-3 installed-rpms 
rhevm-3.5.3.1-1.4.el6ev.noarch                              Tue Aug  4 08:17:55 2015

> 2. exact ovirt-host-deploy version

$ grep ^ovirt-host-deploy installed-rpms 
ovirt-host-deploy-1.3.0-2.el6ev.noarch                      Tue Aug  4 08:17:30 2015
ovirt-host-deploy-java-1.3.0-2.el6ev.noarch                 Tue Aug  4 08:17:30 2015

> 3. /var/cache/ovirt-engine/ovirt-host-deploy.tar - I hope you saved it, if
> not we will not be able to understand what happened in this case as well.

see end of comment 1 for the location of the files.

> need its timestamp (the original one).

Please bear with me while I request that.

> 4. tar -czf /tmp/deploy.tar.gz /usr/share/ovirt-host-deploy

Please bear with me while I request that.

Comment 9 Alon Bar-Lev 2015-08-10 14:01:29 UTC
Created attachment 1061108 [details]
TS.java

Hi,

Can you please download this simple program to /tmp and run:

# javac TS.java
# TZ="UTC" java TS /usr/share/ovirt-host-deploy/interface-3/ > out.txt

The last two lines of output is the max timestamp of all files.

otopi-1.3.0-2             2014-12-06
ovirt-host-deploy-1.3.0-2 2014-12-06

So we expect result to be at that date, unless there is error in the code or there are extra files in this directory that do not belong.

However, we do not expect anything earlier than that for sure.

Comment 10 Julio Entrena Perez 2015-08-10 14:11:08 UTC
Hi Alon,

I run it against the tarball of /usr/share/ovirt-host-deploy that was provided to us and got the following:

$ TZ="UTC" java TS usr/share/ovirt-host-deploy/interface-3/
Checking: usr/share/ovirt-host-deploy/interface-3/.bundled
Checking: usr/share/ovirt-host-deploy/interface-3/otopi-plugins/ovirt-host-deploy/tune/__init__.pyo
[...]
Checking: usr/share/ovirt-host-deploy/interface-3/otopi-plugins/ovirt-host-deploy/vdsm/config.py
1417889443000
Sat Dec 06 18:10:43 UTC 2014

Comment 14 Alon Bar-Lev 2015-08-10 14:27:31 UTC
Created attachment 1061115 [details]
TS.java

Better output to enable us quick response.

Comment 15 Julio Entrena Perez 2015-08-11 08:37:11 UTC
Created attachment 1061347 [details]
Output of TS.java

Output of TS.java from CU.

Last line:
9999999999999 1431421123000 Tue May 12 08:58:43 UTC 2015

Comment 16 Alon Bar-Lev 2015-08-11 09:04:16 UTC
(In reply to Julio Entrena Perez from comment #15)
> Created attachment 1061347 [details]
> Output of TS.java
> 
> Output of TS.java from CU.
> 
> Last line:
> 9999999999999 1431421123000 Tue May 12 08:58:43 UTC 2015

thank you!

so timestamp meets the expectation of otopi-1.3.2-1.el6ev and ovirt-host-deploy-1.3.0-2.el6ev.

now, the question is how before upgrade you had the same timetamp, this may be possible if users have 'yum update' his system while the channel was enabled, so system pulled the new packages as they are not versionlocked and are backward compatible.

ok, found the issue!

otopi was updated and ovirt-host-deploy remained 1.2.5, this caused the timestamp to be updated to the newer package while when the older was updated it was ignored.

will take it from here, than you for your support.

Comment 17 Julio Entrena Perez 2015-08-11 09:08:54 UTC
RPM shows that otopi was indeed updated on 29th July, before the actual upgrade of RHEV-M that happened on 4th August:

$ egrep otopi\|ovirt-host-deploy installed-rpms 
otopi-1.3.2-1.el6ev.noarch                                  Wed Jul 29 11:33:14 2015
otopi-java-1.3.2-1.el6ev.noarch                             Wed Jul 29 11:33:20 2015
ovirt-host-deploy-1.3.0-2.el6ev.noarch                      Tue Aug  4 08:17:30 2015
ovirt-host-deploy-java-1.3.0-2.el6ev.noarch                 Tue Aug  4 08:17:30 2015

Thank you very much Alon.

Comment 19 Pavol Brilla 2015-10-12 08:16:22 UTC
Verified on build 3.6.0-15


/var/cache/ovirt-engine/ovirt-host-deploy.tar is generated before each deploy ( Install/Reinstall )

Comment 22 errata-xmlrpc 2016-03-09 21:10:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0376.html


Note You need to log in before you can comment on or make changes to this bug.