Description of problem: Upgrading 14->15, the `upgrade run` step for compute node fails with: TASK [Debug output for task: Run puppet host configuration for step 4] ********* Friday 19 July 2019 14:01:17 +0000 (0:00:05.583) 0:06:20.818 *********** fatal: [compute-0]: FAILED! => { ... snip ... "<13>Jul 19 14:01:16 puppet-user: Error: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests ON_BOOT]: Could not evaluate: No such file or directory @ rb_sysopen - /etc/sy$ config/libvirt-guests", "<13>Jul 19 14:01:16 puppet-user: Error: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests ON_SHUTDOWN]: Could not evaluate: No such file or directory @ rb_sysopen - /et$ /sysconfig/libvirt-guests", "<13>Jul 19 14:01:16 puppet-user: Error: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests SHUTDOWN_TIMEOUT]: Could not evaluate: No such file or directory @ rb_sysopen $ /etc/sysconfig/libvirt-guests", Installing libvirt-client on the host is a successful workaround, but it's been suggested it may not be the proper solution (https://review.opendev.org/#/c/669122/). Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-10.6.1-0.20190713150434.2871ce0.el8ost.noarch puppet-tripleo-10.4.2-0.20190701160408.ecbec17.el8ost.noarch puppet-nova-14.4.1-0.20190605170411.17663a5.el8ost.noarch Steps to Reproduce: WIP upgrade guide https://gitlab.cee.redhat.com/osp15/osp-upgrade-el8/blob/master/README.md
Jiri, from the attached log we see the norpm messages from the log. Means that puppet-nova tried to install it, but the norpm provider prevents the package from being installed: "<13>Jul 19 14:01:16 puppet-user: Warning: [norpm] Attempting to install libvirt-client but it will not be installed", Do you set EnablePackageInstall for the upgrade? If this is an update/upgrade action EnablePackageInstall should be true, so it doesn't get norpm'd. E.g. if I remove the libvirt-client rpm from a compute and deploy using 'EnablePackageInstall: True' the RPM gets correct installed: Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Executing: '/usr/bin/rpm -q libvirt-client --nosignature --nodigest --qf %{NAME} %|EPOCH?{%{EPOCH}}:{0}| %{VERSION} %{RELEASE} %{ARCH}\\n'", Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Executing: '/usr/bin/rpm -q libvirt-client --nosignature --nodigest --qf %{NAME} %|EPOCH?{%{EPOCH}}:{0}| %{VERSION} %{RELEASE} %{ARCH}\\n --whatprovides'", Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Package[nova-libvirt-guests](provider=yum): Ensuring => present", Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Executing: '/usr/bin/yum -d 0 -e 0 -y install libvirt-client'", Jul 31 11:19:56 compute-0 os-collect-config: "Notice: /Stage[main]/Nova::Compute::Libvirt_guests/Nova::Generic_service[libvirt-guests]/Package[nova-libvirt-guests]/ensure: created", Jul 31 11:19:56 compute-0 os-collect-config: "Info: /Stage[main]/Nova::Compute::Libvirt_guests/Nova::Generic_service[libvirt-guests]/Package[nova-libvirt-guests]: Scheduling refresh of Anchor[nova::install::end]", Jul 31 11:19:56 compute-0 os-collect-config: "Debug: /Stage[main]/Nova::Compute::Libvirt_guests/Nova::Generic_service[libvirt-guests]/Package[nova-libvirt-guests]: The container Nova::Generic_service[libvirt-guests] will propagate my refresh event", Jul 31 11:19:56 compute-0 os-collect-config: "Notice: /Stage[main]/Nova::Deps/Anchor[nova::install::end]: Triggered 'refresh' from 1 events", Jul 31 11:19:56 compute-0 os-collect-config: "Info: /Stage[main]/Nova::Deps/Anchor[nova::install::end]: Scheduling refresh of Anchor[nova::service::begin]", Jul 31 11:19:56 compute-0 os-collect-config: "Debug: /Stage[main]/Nova::Deps/Anchor[nova::install::end]: The container Class[Nova::Deps] will propagate my refresh event", Jul 31 11:19:56 compute-0 os-collect-config: "Notice: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests ON_BOOT]/ensure: created", Jul 31 11:19:56 compute-0 os-collect-config: "Info: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests ON_BOOT]: Scheduling refresh of Service[nova-libvirt-guests]", Jul 31 11:19:56 compute-0 os-collect-config: "Debug: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests ON_BOOT]: The container Class[Nova::Compute::Libvirt_guests] will propagate my refresh event", Jul 31 11:19:56 compute-0 os-collect-config: "Notice: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests ON_SHUTDOWN]/ensure: created", Jul 31 11:19:56 compute-0 os-collect-config: "Info: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests ON_SHUTDOWN]: Scheduling refresh of Service[nova-libvirt-guests]", Jul 31 11:19:56 compute-0 os-collect-config: "Debug: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests ON_SHUTDOWN]: The container Class[Nova::Compute::Libvirt_guests] will propagate my refresh event", Jul 31 11:19:56 compute-0 os-collect-config: "Notice: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests SHUTDOWN_TIMEOUT]/ensure: created", Jul 31 11:19:56 compute-0 os-collect-config: "Info: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests SHUTDOWN_TIMEOUT]: Scheduling refresh of Service[nova-libvirt-guests]", Jul 31 11:19:56 compute-0 os-collect-config: "Debug: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests SHUTDOWN_TIMEOUT]: The container Class[Nova::Compute::Libvirt_guests] will propagate my refresh event", Jul 31 11:19:56 compute-0 os-collect-config: "Notice: /Stage[main]/Nova::Deps/Anchor[nova::service::begin]: Triggered 'refresh' from 1 events", Jul 31 11:19:56 compute-0 os-collect-config: "Info: /Stage[main]/Nova::Deps/Anchor[nova::service::begin]: Scheduling refresh of Service[nova-libvirt-guests]", Jul 31 11:19:56 compute-0 os-collect-config: "Debug: /Stage[main]/Nova::Deps/Anchor[nova::service::begin]: The container Class[Nova::Deps] will propagate my refresh event", Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Executing: '/usr/bin/systemctl is-active libvirt-guests'", Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Executing: '/usr/bin/systemctl is-enabled libvirt-guests'", Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Executing: '/usr/bin/systemctl unmask libvirt-guests'", Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Executing: '/usr/bin/systemctl start libvirt-guests'", Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Executing: '/usr/bin/systemctl enable libvirt-guests'", Jul 31 11:19:56 compute-0 os-collect-config: "Notice: /Stage[main]/Nova::Compute::Libvirt_guests/Nova::Generic_service[libvirt-guests]/Service[nova-libvirt-guests]/ensure: ensure changed 'stopped' to 'running'", Jul 31 11:19:56 compute-0 os-collect-config: "Info: /Stage[main]/Nova::Compute::Libvirt_guests/Nova::Generic_service[libvirt-guests]/Service[nova-libvirt-guests]: Scheduling refresh of Anchor[nova::service::end]", Jul 31 11:19:56 compute-0 os-collect-config: "Debug: /Stage[main]/Nova::Compute::Libvirt_guests/Nova::Generic_service[libvirt-guests]/Service[nova-libvirt-guests]: The container Nova::Generic_service[libvirt-guests] will propagate my refresh event", Jul 31 11:19:56 compute-0 os-collect-config: "Info: /Stage[main]/Nova::Compute::Libvirt_guests/Nova::Generic_service[libvirt-guests]/Service[nova-libvirt-guests]: Unscheduling refresh on Service[nova-libvirt-guests]", Jul 31 11:19:56 compute-0 os-collect-config: "Notice: /Stage[main]/Nova::Deps/Anchor[nova::service::end]: Triggered 'refresh' from 1 events", Jul 31 11:19:56 compute-0 os-collect-config: "Debug: /Stage[main]/Nova::Deps/Anchor[nova::service::end]: The container Class[Nova::Deps] will propagate my refresh event", Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Class[Nova::Deps]: The container Stage[main] will propagate my refresh event", Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Nova::Generic_service[libvirt-guests]: The container Class[Nova::Compute::Libvirt_guests] will propagate my refresh event", Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Class[Nova::Compute::Libvirt_guests]: The container Stage[main] will propagate my refresh event",
Indeed that could be the case, we'll default EnablePackageInstall to true for the duration of the upgrade and hopefully it will solve the issue. Thanks for your investigation.
Patch posted, worked in my testing.
Merged to master, stable/stein backport proposed.
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text. If this bug does not require doc text, please set the 'requires_doc_text' flag to '-'.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0643