Description of problem: After upgrading RHEV-M from 3.4.5 to 3.5.3, attempting to add hosts to RHEV-M fails. Version-Release number of selected component (if applicable): rhevm-backend-3.5.3.1-1.4.el6ev How reproducible: Unknown. Steps to Reproduce: 1. Upgrade RHEV-M. 2. Try to add a new host. 3. Actual results: Host fails to be added: 2015-08-05 16:33:22,561 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-17) [2c3d77b7] START, AddVdsVDSCommand(HostName = hostname, HostId = b5e65b80-a85c-47c0-8394-ffd6157c0b4d), log id: 6b8b3f41 2015-08-05 16:33:22,561 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-17) [2c3d77b7] AddVds - entered , starting logic to add VDS b5e65b80-a85c-47c0-8394-ffd6157c0b4d [...] 2015-08-05 16:33:29,235 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 74afa17c, Call Stack: null, Custom Event ID: -1, Message: Installing Host hostname. Logs at host located at: '/tmp/ovirt-host-deploy-20150805163326-pf6cid.log'. 2015-08-05 16:33:29,279 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host 172.20.46.118 reports unique id 30313436-3631-5A43-4A34-31323 0323138 2015-08-05 16:33:29,300 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 30313436-3631-5A43-4A34-313230323138 to Host 172.20.46.118 2015-08-05 16:33:29,394 ERROR [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Error during deploy dialog: java.lang.NullPointerException at org.ovirt.engine.core.bll.VdsDeploy$37.call(VdsDeploy.java:580) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy$37.call(VdsDeploy.java:579) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy._nextCustomizationEntry(VdsDeploy.java:668) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy._threadMain(VdsDeploy.java:873) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy.access$2000(VdsDeploy.java:83) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy$51.run(VdsDeploy.java:969) [bll.jar:] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79] 2015-08-05 16:33:29,408 ERROR [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-7-thread-33) [74afa17c] Error during host 172.20.46.118 install: java.lang.NullPointerException at org.ovirt.engine.core.bll.VdsDeploy$37.call(VdsDeploy.java:580) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy$37.call(VdsDeploy.java:579) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy._nextCustomizationEntry(VdsDeploy.java:668) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy._threadMain(VdsDeploy.java:873) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy.access$2000(VdsDeploy.java:83) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy$51.run(VdsDeploy.java:969) [bll.jar:] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79] 2015-08-05 16:33:29,413 ERROR [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-7-thread-33) [74afa17c] Installation 172.20.46.118: null 2015-08-05 16:33:29,453 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-33) [74afa17c] Correlation ID: 74afa17c, Call Stack: null, Custom Event ID: -1, Message: Failed to install Host hostname. <UNKNOWN>. 2015-08-05 16:33:29,454 ERROR [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-7-thread-33) [74afa17c] Error during host 172.20.46.118 install, prefering first exception: java.lang.NullPointerException at org.ovirt.engine.core.bll.VdsDeploy$37.call(VdsDeploy.java:580) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy$37.call(VdsDeploy.java:579) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy._nextCustomizationEntry(VdsDeploy.java:668) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy._threadMain(VdsDeploy.java:873) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy.access$2000(VdsDeploy.java:83) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy$51.run(VdsDeploy.java:969) [bll.jar:] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79] 2015-08-05 16:33:29,454 ERROR [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-33) [74afa17c] Host installation failed for host b5e65b80-a85c-47c0-8394-ffd6157c0b4d, hostname.: java.lang.NullPointerException at org.ovirt.engine.core.bll.VdsDeploy$37.call(VdsDeploy.java:580) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy$37.call(VdsDeploy.java:579) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy._nextCustomizationEntry(VdsDeploy.java:668) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy._threadMain(VdsDeploy.java:873) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy.access$2000(VdsDeploy.java:83) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy$51.run(VdsDeploy.java:969) [bll.jar:] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79] 2015-08-05 16:33:29,482 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-33) [74afa17c] START, SetVdsStatusVDSCommand(HostName = hostname, HostId = b5e65b80-a85c-47c0-8394-ffd6157c0b4d, status=InstallFailed, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 70909605 2015-08-05 16:33:29,501 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-33) [74afa17c] FINISH, SetVdsStatusVDSCommand, log id: 70909605 2015-08-05 16:33:29,522 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-33) [74afa17c] Correlation ID: 74afa17c, Job ID: fb04dfb2-9537-4691-bca0-13c35f0f5f79, Call Stack: null, Custom Event ID: -1, Message: Host hostname installation failed. Please refer to /var/log/ovirt-engine/engine.log and log logs under /var/log/ovirt-engine/host-deploy/ for further details.. Expected results: Host is added successfully. Additional info: After manually deleting file /var/cache/ovirt-engine/ovirt-host-deploy.tar from RHEV-M, host was added successfully.
please provide: 1. exact engine version 2. exact ovirt-host-deploy version 3. /var/cache/ovirt-engine/ovirt-host-deploy.tar - I hope you saved it, if not we will not be able to understand what happened in this case as well. 4. tar -czf /tmp/deploy.tar.gz /usr/share/ovirt-host-deploy thanks!
(In reply to Alon Bar-Lev from comment #3) > please provide: > > 1. exact engine version > 2. exact ovirt-host-deploy version > 3. /var/cache/ovirt-engine/ovirt-host-deploy.tar - I hope you saved it, if > not we will not be able to understand what happened in this case as well. need its timestamp (the original one). > 4. tar -czf /tmp/deploy.tar.gz /usr/share/ovirt-host-deploy > > thanks!
(In reply to Alon Bar-Lev from comment #3) > please provide: > > 1. exact engine version $ grep rhevm-3 installed-rpms rhevm-3.5.3.1-1.4.el6ev.noarch Tue Aug 4 08:17:55 2015 > 2. exact ovirt-host-deploy version $ grep ^ovirt-host-deploy installed-rpms ovirt-host-deploy-1.3.0-2.el6ev.noarch Tue Aug 4 08:17:30 2015 ovirt-host-deploy-java-1.3.0-2.el6ev.noarch Tue Aug 4 08:17:30 2015 > 3. /var/cache/ovirt-engine/ovirt-host-deploy.tar - I hope you saved it, if > not we will not be able to understand what happened in this case as well. see end of comment 1 for the location of the files. > need its timestamp (the original one). Please bear with me while I request that. > 4. tar -czf /tmp/deploy.tar.gz /usr/share/ovirt-host-deploy Please bear with me while I request that.
Created attachment 1061108 [details] TS.java Hi, Can you please download this simple program to /tmp and run: # javac TS.java # TZ="UTC" java TS /usr/share/ovirt-host-deploy/interface-3/ > out.txt The last two lines of output is the max timestamp of all files. otopi-1.3.0-2 2014-12-06 ovirt-host-deploy-1.3.0-2 2014-12-06 So we expect result to be at that date, unless there is error in the code or there are extra files in this directory that do not belong. However, we do not expect anything earlier than that for sure.
Hi Alon, I run it against the tarball of /usr/share/ovirt-host-deploy that was provided to us and got the following: $ TZ="UTC" java TS usr/share/ovirt-host-deploy/interface-3/ Checking: usr/share/ovirt-host-deploy/interface-3/.bundled Checking: usr/share/ovirt-host-deploy/interface-3/otopi-plugins/ovirt-host-deploy/tune/__init__.pyo [...] Checking: usr/share/ovirt-host-deploy/interface-3/otopi-plugins/ovirt-host-deploy/vdsm/config.py 1417889443000 Sat Dec 06 18:10:43 UTC 2014
Created attachment 1061115 [details] TS.java Better output to enable us quick response.
Created attachment 1061347 [details] Output of TS.java Output of TS.java from CU. Last line: 9999999999999 1431421123000 Tue May 12 08:58:43 UTC 2015
(In reply to Julio Entrena Perez from comment #15) > Created attachment 1061347 [details] > Output of TS.java > > Output of TS.java from CU. > > Last line: > 9999999999999 1431421123000 Tue May 12 08:58:43 UTC 2015 thank you! so timestamp meets the expectation of otopi-1.3.2-1.el6ev and ovirt-host-deploy-1.3.0-2.el6ev. now, the question is how before upgrade you had the same timetamp, this may be possible if users have 'yum update' his system while the channel was enabled, so system pulled the new packages as they are not versionlocked and are backward compatible. ok, found the issue! otopi was updated and ovirt-host-deploy remained 1.2.5, this caused the timestamp to be updated to the newer package while when the older was updated it was ignored. will take it from here, than you for your support.
RPM shows that otopi was indeed updated on 29th July, before the actual upgrade of RHEV-M that happened on 4th August: $ egrep otopi\|ovirt-host-deploy installed-rpms otopi-1.3.2-1.el6ev.noarch Wed Jul 29 11:33:14 2015 otopi-java-1.3.2-1.el6ev.noarch Wed Jul 29 11:33:20 2015 ovirt-host-deploy-1.3.0-2.el6ev.noarch Tue Aug 4 08:17:30 2015 ovirt-host-deploy-java-1.3.0-2.el6ev.noarch Tue Aug 4 08:17:30 2015 Thank you very much Alon.
Verified on build 3.6.0-15 /var/cache/ovirt-engine/ovirt-host-deploy.tar is generated before each deploy ( Install/Reinstall )
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-0376.html