Bug 1731505 - OSP 14->15: compute node upgrade fails on editing libvirt-guests config [NEEDINFO]
Summary: OSP 14->15: compute node upgrade fails on editing libvirt-guests config
Keywords:
Status: ON_QA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Sergii Golovatiuk
QA Contact: Jose Luis Franco
URL:
Whiteboard:
Depends On: 1803434
Blocks: 1727807
TreeView+ depends on / blocked
 
Reported: 2019-07-19 15:05 UTC by Jiri Stransky
Modified: 2020-02-26 11:37 UTC (History)
9 users (show)

Fixed In Version: openstack-tripleo-heat-templates-10.6.1-0.20190909163923.999c846.el8ost.noarch
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
amcleod: needinfo? (sgolovat)


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Launchpad 1837228 None None None 2019-07-31 10:34:45 UTC
OpenStack gerrit 679003 'None' MERGED Set EnablePackageInstall to true by default for in-place OS upgrade 2020-02-24 14:13:46 UTC

Description Jiri Stransky 2019-07-19 15:05:20 UTC
Description of problem:

Upgrading 14->15, the `upgrade run` step for compute node fails with:

TASK [Debug output for task: Run puppet host configuration for step 4] *********
Friday 19 July 2019  14:01:17 +0000 (0:00:05.583)       0:06:20.818 *********** 
fatal: [compute-0]: FAILED! => {

... snip ...

        "<13>Jul 19 14:01:16 puppet-user: Error: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests ON_BOOT]: Could not evaluate: No such file or directory @ rb_sysopen - /etc/sy$
config/libvirt-guests",
        "<13>Jul 19 14:01:16 puppet-user: Error: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests ON_SHUTDOWN]: Could not evaluate: No such file or directory @ rb_sysopen - /et$
/sysconfig/libvirt-guests",
        "<13>Jul 19 14:01:16 puppet-user: Error: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests SHUTDOWN_TIMEOUT]: Could not evaluate: No such file or directory @ rb_sysopen $
 /etc/sysconfig/libvirt-guests",


Installing libvirt-client on the host is a successful workaround, but it's been suggested it may not be the proper solution (https://review.opendev.org/#/c/669122/).


Version-Release number of selected component (if applicable):

openstack-tripleo-heat-templates-10.6.1-0.20190713150434.2871ce0.el8ost.noarch
puppet-tripleo-10.4.2-0.20190701160408.ecbec17.el8ost.noarch
puppet-nova-14.4.1-0.20190605170411.17663a5.el8ost.noarch

Steps to Reproduce:

WIP upgrade guide https://gitlab.cee.redhat.com/osp15/osp-upgrade-el8/blob/master/README.md

Comment 2 Martin Schuppert 2019-07-31 13:02:36 UTC
Jiri, from the attached log we see the norpm messages from the log. Means that puppet-nova tried to install it, but 
the norpm provider prevents the package from being installed:

        "<13>Jul 19 14:01:16 puppet-user: Warning: [norpm] Attempting to install libvirt-client but it will not be installed",

Do you set EnablePackageInstall for the upgrade? If this is an update/upgrade action EnablePackageInstall should be true, so it doesn't get norpm'd.

E.g. if I remove the libvirt-client rpm from a compute and deploy using 'EnablePackageInstall: True' the RPM gets correct installed:

Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Executing: '/usr/bin/rpm -q libvirt-client --nosignature --nodigest --qf %{NAME} %|EPOCH?{%{EPOCH}}:{0}| %{VERSION} %{RELEASE} %{ARCH}\\n'",
Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Executing: '/usr/bin/rpm -q libvirt-client --nosignature --nodigest --qf %{NAME} %|EPOCH?{%{EPOCH}}:{0}| %{VERSION} %{RELEASE} %{ARCH}\\n --whatprovides'",
Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Package[nova-libvirt-guests](provider=yum): Ensuring => present",
Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Executing: '/usr/bin/yum -d 0 -e 0 -y install libvirt-client'",
Jul 31 11:19:56 compute-0 os-collect-config: "Notice: /Stage[main]/Nova::Compute::Libvirt_guests/Nova::Generic_service[libvirt-guests]/Package[nova-libvirt-guests]/ensure: created",
Jul 31 11:19:56 compute-0 os-collect-config: "Info: /Stage[main]/Nova::Compute::Libvirt_guests/Nova::Generic_service[libvirt-guests]/Package[nova-libvirt-guests]: Scheduling refresh of Anchor[nova::install::end]",
Jul 31 11:19:56 compute-0 os-collect-config: "Debug: /Stage[main]/Nova::Compute::Libvirt_guests/Nova::Generic_service[libvirt-guests]/Package[nova-libvirt-guests]: The container Nova::Generic_service[libvirt-guests] will propagate my refresh event",
Jul 31 11:19:56 compute-0 os-collect-config: "Notice: /Stage[main]/Nova::Deps/Anchor[nova::install::end]: Triggered 'refresh' from 1 events",
Jul 31 11:19:56 compute-0 os-collect-config: "Info: /Stage[main]/Nova::Deps/Anchor[nova::install::end]: Scheduling refresh of Anchor[nova::service::begin]",
Jul 31 11:19:56 compute-0 os-collect-config: "Debug: /Stage[main]/Nova::Deps/Anchor[nova::install::end]: The container Class[Nova::Deps] will propagate my refresh event",
Jul 31 11:19:56 compute-0 os-collect-config: "Notice: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests ON_BOOT]/ensure: created",
Jul 31 11:19:56 compute-0 os-collect-config: "Info: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests ON_BOOT]: Scheduling refresh of Service[nova-libvirt-guests]",
Jul 31 11:19:56 compute-0 os-collect-config: "Debug: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests ON_BOOT]: The container Class[Nova::Compute::Libvirt_guests] will propagate my refresh event",
Jul 31 11:19:56 compute-0 os-collect-config: "Notice: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests ON_SHUTDOWN]/ensure: created",
Jul 31 11:19:56 compute-0 os-collect-config: "Info: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests ON_SHUTDOWN]: Scheduling refresh of Service[nova-libvirt-guests]",
Jul 31 11:19:56 compute-0 os-collect-config: "Debug: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests ON_SHUTDOWN]: The container Class[Nova::Compute::Libvirt_guests] will propagate my refresh event",
Jul 31 11:19:56 compute-0 os-collect-config: "Notice: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests SHUTDOWN_TIMEOUT]/ensure: created",
Jul 31 11:19:56 compute-0 os-collect-config: "Info: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests SHUTDOWN_TIMEOUT]: Scheduling refresh of Service[nova-libvirt-guests]",
Jul 31 11:19:56 compute-0 os-collect-config: "Debug: /Stage[main]/Nova::Compute::Libvirt_guests/File_line[/etc/sysconfig/libvirt-guests SHUTDOWN_TIMEOUT]: The container Class[Nova::Compute::Libvirt_guests] will propagate my refresh event",
Jul 31 11:19:56 compute-0 os-collect-config: "Notice: /Stage[main]/Nova::Deps/Anchor[nova::service::begin]: Triggered 'refresh' from 1 events",
Jul 31 11:19:56 compute-0 os-collect-config: "Info: /Stage[main]/Nova::Deps/Anchor[nova::service::begin]: Scheduling refresh of Service[nova-libvirt-guests]",
Jul 31 11:19:56 compute-0 os-collect-config: "Debug: /Stage[main]/Nova::Deps/Anchor[nova::service::begin]: The container Class[Nova::Deps] will propagate my refresh event",
Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Executing: '/usr/bin/systemctl is-active libvirt-guests'",
Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Executing: '/usr/bin/systemctl is-enabled libvirt-guests'",
Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Executing: '/usr/bin/systemctl unmask libvirt-guests'",
Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Executing: '/usr/bin/systemctl start libvirt-guests'",
Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Executing: '/usr/bin/systemctl enable libvirt-guests'",
Jul 31 11:19:56 compute-0 os-collect-config: "Notice: /Stage[main]/Nova::Compute::Libvirt_guests/Nova::Generic_service[libvirt-guests]/Service[nova-libvirt-guests]/ensure: ensure changed 'stopped' to 'running'",
Jul 31 11:19:56 compute-0 os-collect-config: "Info: /Stage[main]/Nova::Compute::Libvirt_guests/Nova::Generic_service[libvirt-guests]/Service[nova-libvirt-guests]: Scheduling refresh of Anchor[nova::service::end]",
Jul 31 11:19:56 compute-0 os-collect-config: "Debug: /Stage[main]/Nova::Compute::Libvirt_guests/Nova::Generic_service[libvirt-guests]/Service[nova-libvirt-guests]: The container Nova::Generic_service[libvirt-guests] will propagate my refresh event",
Jul 31 11:19:56 compute-0 os-collect-config: "Info: /Stage[main]/Nova::Compute::Libvirt_guests/Nova::Generic_service[libvirt-guests]/Service[nova-libvirt-guests]: Unscheduling refresh on Service[nova-libvirt-guests]",
Jul 31 11:19:56 compute-0 os-collect-config: "Notice: /Stage[main]/Nova::Deps/Anchor[nova::service::end]: Triggered 'refresh' from 1 events",
Jul 31 11:19:56 compute-0 os-collect-config: "Debug: /Stage[main]/Nova::Deps/Anchor[nova::service::end]: The container Class[Nova::Deps] will propagate my refresh event",
Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Class[Nova::Deps]: The container Stage[main] will propagate my refresh event",
Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Nova::Generic_service[libvirt-guests]: The container Class[Nova::Compute::Libvirt_guests] will propagate my refresh event",
Jul 31 11:19:56 compute-0 os-collect-config: "Debug: Class[Nova::Compute::Libvirt_guests]: The container Stage[main] will propagate my refresh event",

Comment 3 Jiri Stransky 2019-08-01 13:02:55 UTC
Indeed that could be the case, we'll default EnablePackageInstall to true for the duration of the upgrade and hopefully it will solve the issue. Thanks for your investigation.

Comment 5 Jiri Stransky 2019-08-14 15:11:15 UTC
Patch posted, worked in my testing.

Comment 7 Jiri Stransky 2019-08-28 12:28:39 UTC
Merged to master, stable/stein backport proposed.

Comment 13 Alex McLeod 2020-02-19 12:49:01 UTC
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to '-'.


Note You need to log in before you can comment on or make changes to this bug.