Created attachment 880737 [details] engine.log Description of problem: while trying to re-install a host with rhev-h using a different iso of an older rhevh version, install failed because engine (3.4) was trying to read from file: /usr/share/rhev-hypervisor/version-latest-6.txt checking the /usr/share/rhev-hypervisor/ directory there was no version-latest-6.txt file but version.txt which contained the rhevh version for the version I was trying to install. after changing the name of version.txt to version-latest-6.txt installation worked Steps to Reproduce: 1. add host with rhevh to 3.4 engine 2. yum localinstall rhev-h 20131126.0.3.2.el6_5 on the engine machine 3. put host to maintenance 4. create 3.2 datacenter and cluster 5. edit host and change it to 3.2 dc-cluster 5. re-install host and choose 20131126.0.3.2.iso Actual results: installation fails, host moves to non-operational state Expected results: re-installation succedes and host moves to up state. Additional info: from engine.log: 2014-03-31 11:02:47,583 ERROR [org.ovirt.engine.core.bll.GetoVirtISOsQuery] (ajp-/127.0.0.1:8702-2) Query GetoVirtISOsQuery failed. Exception message is null : java.lang.NullPointerException: java.lang.NullPointerException 2014-03-31 11:02:47,785 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-67) [34236216] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 4a6e622b-856c-49cb-ac26-c4baa71fca84 Type: VDS 2014-03-31 11:02:47,841 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-67) [6c7379d3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 4a6e622b-856c-49cb-ac26-c4baa71fca84 Type: VDS 2014-03-31 11:02:52,604 ERROR [org.ovirt.engine.core.bll.GetoVirtISOsQuery] (ajp-/127.0.0.1:8702-6) Failed to open version file /usr/share/rhev-hypervisor/version-latest-6.txt with error FileNotFoundException: /usr/share/rhev-hypervisor/version-latest-6.txt (No such file or directory) 2014-03-31 11:03:00,305 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Warning, Low disk space. Host monique-vds04 has less than 1000 MB of free space left on: /tmp, /var/run/vdsm/. 2014-03-31 11:03:02,463 ERROR [org.ovirt.engine.core.bll.GetoVirtISOsQuery] (ajp-/127.0.0.1:8702-1) Failed to open version file /usr/share/rhev-hypervisor/version-latest-6.txt with error FileNotFoundException: /usr/share/rhev-hypervisor/version-latest-6.txt (No such file or directory) 2014-03-31 11:03:45,562 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-5-thread-35) [40d6ed8a] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 4a6e622b-856c-49cb-ac26-c4baa71fca84 Type: VDS
I guess this is due to bug#979231, something was changed in the format we expect?
Although, I could upgrade, I saw the error message reported in engine.log too, (In reply to Alon Bar-Lev from comment #1) > I guess this is due to bug#979231, something was changed in the format we expect? Looks like the new regex expected a format with a single link to iso filename and in this case we had 3 files pointing to the same iso.
It seems as though the rhev-h update bug has more to it then the version.txt file name. rhevh upgrade failed with this patch as well as can be seen in this test run: http://jenkins-ci.eng.lab.tlv.redhat.com/view/0%20Unstable%203.4/job/rhevm_3.4_automation_coretools_two_hosts_rhevh_restapi_hosts_nfs_rest_factory/33/ the relevant entries in the engine.log for this case: 2014-04-29 17:00:19,258 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (ajp-/127.0.0.1:8702-1) [3de8ae44] Lock Acquired to object EngineLock [exclusiveLocks= key: 24c97fea-12b5-40dc-b90f-030341c888a3 value: VDS , sharedLocks= ] 2014-04-29 17:00:19,310 WARN [org.ovirt.engine.core.bll.InstallVdsCommand] (ajp-/127.0.0.1:8702-1) [3de8ae44] CanDoAction of action InstallVds failed. Reasons:VDS_CANNOT_UPGRADE_BETWEEN_MAJOR_VERSION,$IsoVersion 6 2014-04-29 17:00:19,311 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (ajp-/127.0.0.1:8702-1) [3de8ae44] Lock freed to object EngineLock [exclusiveLocks= key: 24c97fea-12b5-40dc-b90f-030341c888a3 value: VDS , sharedLocks= ] 2014-04-29 17:00:19,314 ERROR [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-1) [3de8ae44] Installation/upgrade of Host 24c97fea-12b5-40dc-b90f-030341c888a3,cinteg26.ci.lab.tlv.redhat.com failed due to: Cannot upgrade Host. Host version is not compatible with selected ISO version. Please select an ISO with major version 6.x. 2014-04-29 17:00:19,357 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-1) [3de8ae44] Command [id=a3562da0-4421-4b39-8401-690a71ba77eb]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.VdsDynamic; snapshot: EntityStatusSnapshot [id=24c97fea-12b5-40dc-b90f-030341c888a3, status=Maintenance]. 2014-04-29 17:00:19,396 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-1) [3de8ae44] Command [id=a3562da0-4421-4b39-8401-690a71ba77eb]: Compensating CHANGED_ENTITY of org.ovirt.engine.core.common.businessentities.VdsStatic; snapshot: id=24c97fea-12b5-40dc-b90f-030341c888a3. 2014-04-29 17:00:19,462 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [3de8ae44] Correlation ID: 3de8ae44, Call Stack: null, Custom Event ID: -1, Message: Failed to update Host cinteg26.ci.lab.tlv.redhat.com (User: admin). 2014-04-29 17:00:19,462 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8702-1) Operation Failed: [Cannot upgrade Host. Host version is not compatible with selected ISO version. Please select an ISO with major version 6.x.]
Form a RHEV-H perspective I can say that the file where we keep the version informations wasn't changed recently. Was there something changed on the Engine side to look in different files?
Hi, (In reply to sefi litmanovich from comment #4) > It seems as though the rhev-h update bug has more to it then the version.txt > file name. > rhevh upgrade failed with this patch as well as can be seen in this test run: > http://jenkins-ci.eng.lab.tlv.redhat.com/view/0%20Unstable%203.4/job/rhevm_3.4_automation_coretools_two_hosts_rhevh_restapi_hosts_nfs_rest_factory/33/ > > the relevant entries in the engine.log for this case: > > 2014-04-29 17:00:19,258 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] > (ajp-/127.0.0.1:8702-1) [3de8ae44] Lock Acquired to object EngineLock > [exclusiveLocks= key: 24c97fea-12b5-40dc-b90f-030341c888a3 value: VDS > , sharedLocks= ] > 2014-04-29 17:00:19,310 WARN [org.ovirt.engine.core.bll.InstallVdsCommand] > (ajp-/127.0.0.1:8702-1) [3de8ae44] CanDoAction of action InstallVds failed. > Reasons:VDS_CANNOT_UPGRADE_BETWEEN_MAJOR_VERSION,$IsoVersion 6 > 2014-04-29 17:00:19,311 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] > (ajp-/127.0.0.1:8702-1) [3de8ae44] Lock freed to object EngineLock > [exclusiveLocks= key: 24c97fea-12b5-40dc-b90f-030341c888a3 value: VDS > , sharedLocks= ] > 2014-04-29 17:00:19,314 ERROR [org.ovirt.engine.core.bll.UpdateVdsCommand] > (ajp-/127.0.0.1:8702-1) [3de8ae44] Installation/upgrade of Host > 24c97fea-12b5-40dc-b90f-030341c888a3,cinteg26.ci.lab.tlv.redhat.com failed > due to: Cannot upgrade Host. Host version is not compatible with selected > ISO version. Please select an ISO with major version 6.x. > 2014-04-29 17:00:19,357 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] > (ajp-/127.0.0.1:8702-1) [3de8ae44] Command > [id=a3562da0-4421-4b39-8401-690a71ba77eb]: Compensating CHANGED_STATUS_ONLY > of org.ovirt.engine.core.common.businessentities.VdsDynamic; snapshot: > EntityStatusSnapshot [id=24c97fea-12b5-40dc-b90f-030341c888a3, > status=Maintenance]. > 2014-04-29 17:00:19,396 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] > (ajp-/127.0.0.1:8702-1) [3de8ae44] Command > [id=a3562da0-4421-4b39-8401-690a71ba77eb]: Compensating CHANGED_ENTITY of > org.ovirt.engine.core.common.businessentities.VdsStatic; snapshot: > id=24c97fea-12b5-40dc-b90f-030341c888a3. > 2014-04-29 17:00:19,462 INFO > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (ajp-/127.0.0.1:8702-1) [3de8ae44] Correlation ID: 3de8ae44, Call Stack: > null, Custom Event ID: -1, Message: Failed to update Host > cinteg26.ci.lab.tlv.redhat.com (User: admin). > 2014-04-29 17:00:19,462 ERROR > [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] > (ajp-/127.0.0.1:8702-1) Operation Failed: [Cannot upgrade Host. Host version > is not compatible with selected ISO version. Please select an ISO with major > version 6.x.] This is a different bug, please open a specific bug for it and assign to me. Additionally: - Attach the engine.log and more details which isos of rhev-h you used to test the upgrade. - If possible attach the rest api script automatic job that is reporting this error, so I can reproduce locally - What happens if you try to upgrade via Engine Admin Page instead of using automatic job?
Verified with rhevm-3.4.0-0.16.rc.el6ev.noarch. new bug regarding the rhev-h re-installation issue was opened: https://bugzilla.redhat.com/show_bug.cgi?id=1093303
Closing as part of 3.4.0