Bug 2101481

Summary: abrt was removed in RHEL 8.6 and causes HE deployments to fail it upgraded from RHEL 8.5 and earlier.
Product: Red Hat Enterprise Virtualization Manager Reporter: amashah
Component: ovirt-ansible-collectionAssignee: Marcin Sobczyk <msobczyk>
Status: CLOSED ERRATA QA Contact: Nikolai Sednev <nsednev>
Severity: high Docs Contact:
Priority: high    
Version: 4.5.0CC: arachman, didi, emarcus, mperina, msobczyk, pbrilla
Target Milestone: ovirt-4.5.2Keywords: Triaged, ZStream
Target Release: 4.5.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovirt-ansible-collection-2.1.1 Doc Type: Bug Fix
Doc Text:
The handling of core dumps during upgrade from previous Red Hat Virtualization versions to RHV 4.4 SP1 batch 1 has been fixed.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2022-09-08 11:29:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description amashah 2022-06-27 15:13:35 UTC 
Description of problem:
Checking RHEL 8.6 release notes, apparently abrt was deprecated:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/8.6_release_notes/index

~~~
The ABRT tool has been deprecated

The Automatic Bug Reporting Tool (ABRT) for detecting and reporting application crashes has been deprecated in RHEL 8. As a replacement, use the systemd-coredump tool to log and store core dumps, which are automatically generated files after a program crashes.
~~~


In RHV-H 4.5.0, abrt package is not installed, as a result of above.


In HE deployment, we do:

~~~
  - name: Verify abrt-action-save-package-data file
    stat:
      path: /etc/abrt/abrt-action-save-package-data.conf
    register: abrt_exists
  - name: Restore initial abrt config files
    copy:
      remote_src: true
      src: "{{ item.src }}"
      dest: "{{ item.dest }}"
      mode: preserve
    with_items:
      - {
        src: /usr/share/abrt/conf.d/abrt-action-save-package-data.conf,
        dest: /etc/abrt/abrt-action-save-package-data.conf
      }
      - {
        src: /usr/share/abrt/conf.d/abrt.conf,
        dest: /etc/abrt/abrt.conf
      }
      - {
        src: /usr/share/abrt/conf.d/plugins/CCpp.conf,
        dest: /etc/abrt/plugins/CCpp.conf
      }
      - {
        src: /usr/share/abrt/conf.d/plugins/vmcore.conf,
        dest: /etc/abrt/plugins/vmcore.conf
      }
    when: abrt_exists.stat.exists
~~~


If the host was upgraded from RHV-H based on RHEL 8.5 or earlier, then there is a remnant of abrt in /etc/abrt/abrt-action-save-package-data.conf so it tries to copy the files above.

They source files do not exist in 8.6 based hosts, so it fails with:

~~~
2022-06-27 13:36:51,627+0530 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 TASK [ovirt.ovirt.hosted_engine_setup : Verify abrt-action-save-package-data file]
2022-06-27 13:36:52,228+0530 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 ok: [localhost]


2022-06-27 13:36:52,329+0530 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 TASK [ovirt.ovirt.hosted_engine_setup : Restore initial abrt config files]
2022-06-27 13:36:53,431+0530 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 {'results': [{'failed': True, 'msg': 'Source /usr/share/abrt/conf.d/abrt-action-save-package-data.conf not found', 'invocation': {'module_args': {'remote_src': True, 'src': '/usr/share/abrt/conf.d/abrt-action-save-package-data.conf', 'dest': '/etc/abrt/abrt-action-save-package-data.conf', 'mode': 'preserve', 'backup': False, 'force': True, 'follow': False, 'unsafe_writes': False, '_original_basename': None, 'content': None, 'validate': None, 'directory_mode': None, 'local_follow': None, 'checksum': None, 'owner': None, 'group': None, 'seuser': None, 'serole': None, 'selevel': None, 'setype': None, 'attributes': None}}, '_ansible_no_log': False, 'changed': False, 'item': {'src': '/usr/share/abrt/conf.d/abrt-action-save-package-data.conf', 'dest': '/etc/abrt/abrt-action-save-package-data.conf'}, 'ansible_loop_var': 'item', '_ansible_item_label': {'src': '/usr/share/abrt/conf.d/abrt-action-save-package-data.conf', 'dest': '/etc/abrt/abrt-action-save-package-data.conf'}}, {'failed': True, 'msg': 'Source /usr/share/abrt/conf.d/abrt.conf not found', 'invocation': {'module_args': {'remote_src': True, 'src': '/usr/share/abrt/conf.d/abrt.conf', 'dest': '/etc/abrt/abrt.conf', 'mode': 'preserve', 'backup': False, 'force': True, 'follow': False, 'unsafe_writes': False, '_original_basename': None, 'content': None, 'validate': None, 'directory_mode': None, 'local_follow': None, 'checksum': None, 'owner': None, 'group': None, 'seuser': None, 'serole': None, 'selevel': None, 'setype': None, 'attributes': None}}, '_ansible_no_log': False, 'changed': False, 'item': {'src': '/usr/share/abrt/conf.d/abrt.conf', 'dest': '/etc/abrt/abrt.conf'}, 'ansible_loop_var': 'item', '_ansible_item_label': {'src': '/usr/share/abrt/conf.d/abrt.conf', 'dest': '/etc/abrt/abrt.conf'}}, {'failed': True, 'msg': 'Source /usr/share/abrt/conf.d/plugins/CCpp.conf not found', 'invocation': {'module_args': {'remote_src': True, 'src': '/usr/share/abrt/conf.d/plugins/CCpp.conf', 'dest': '/etc/abrt/plugins/CCpp.conf', 'mode': 'preserve', 'backup': False, 'force': True, 'follow': False, 'unsafe_writes': False, '_original_basename': None, 'content': None, 'validate': None, 'directory_mode': None, 'local_follow': None, 'checksum': None, 'owner': None, 'group': None, 'seuser': None, 'serole': None, 'selevel': None, 'setype': None, 'attributes': None}}, '_ansible_no_log': False, 'changed': False, 'item': {'src': '/usr/share/abrt/conf.d/plugins/CCpp.conf', 'dest': '/etc/abrt/plugins/CCpp.conf'}, 'ansible_loop_var': 'item', '_ansible_item_label': {'src': '/usr/share/abrt/conf.d/plugins/CCpp.conf', 'dest': '/etc/abrt/plugins/CCpp.conf'}}, {'failed': True, 'msg': 'Source /usr/share/abrt/conf.d/plugins/vmcore.conf not found', 'invocation': {'module_args': {'remote_src': True, 'src': '/usr/share/abrt/conf.d/plugins/vmcore.conf', 'dest': '/etc/abrt/plugins/vmcore.conf', 'mode': 'preserve', 'backup': False, 'force': True, 'follow': False, 'unsafe_writes': False, '_original_basename': None, 'content': None, 'validate': None, 'directory_mode': None, 'local_follow': None, 'checksum': None, 'owner': None, 'group': None, 'seuser': None, 'serole': None, 'selevel': None, 'setype': None, 'attributes': None}}, '_ansible_no_log': False, 'changed': False, 'item': {'src': '/usr/share/abrt/conf.d/plugins/vmcore.conf', 'dest': '/etc/abrt/plugins/vmcore.conf'}, 'ansible_loop_var': 'item', '_ansible_item_label': {'src': '/usr/share/abrt/conf.d/plugins/vmcore.conf', 'dest': '/etc/abrt/plugins/vmcore.conf'}}], 'msg': 'One or more items failed', 'changed': False}
2022-06-27 13:36:53,532+0530 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113 {'failed': True, 'msg': 'Source /usr/share/abrt/conf.d/abrt-action-save-package-data.conf not found', 'invocation': {'module_args': {'remote_src': True, 'src': '/usr/share/abrt/conf.d/abrt-action-save-package-data.conf', 'dest': '/etc/abrt/abrt-action-save-package-data.conf', 'mode': 'preserve', 'backup': False, 'force': True, 'follow': False, 'unsafe_writes': False, '_original_basename': None, 'content': None, 'validate': None, 'directory_mode': None, 'local_follow': None, 'checksum': None, 'owner': None, 'group': None, 'seuser': None, 'serole': None, 'selevel': None, 'setype': None, 'attributes': None}}, '_ansible_no_log': False, 'changed': False, 'item': {'src': '/usr/share/abrt/conf.d/abrt-action-save-package-data.conf', 'dest': '/etc/abrt/abrt-action-save-package-data.conf'}, 'ansible_loop_var': 'item', '_ansible_item_label': {'src': '/usr/share/abrt/conf.d/abrt-action-save-package-data.conf', 'dest': '/etc/abrt/abrt-action-save-package-data.conf'}}
2022-06-27 13:36:53,633+0530 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113 {'failed': True, 'msg': 'Source /usr/share/abrt/conf.d/abrt.conf not found', 'invocation': {'module_args': {'remote_src': True, 'src': '/usr/share/abrt/conf.d/abrt.conf', 'dest': '/etc/abrt/abrt.conf', 'mode': 'preserve', 'backup': False, 'force': True, 'follow': False, 'unsafe_writes': False, '_original_basename': None, 'content': None, 'validate': None, 'directory_mode': None, 'local_follow': None, 'checksum': None, 'owner': None, 'group': None, 'seuser': None, 'serole': None, 'selevel': None, 'setype': None, 'attributes': None}}, '_ansible_no_log': False, 'changed': False, 'item': {'src': '/usr/share/abrt/conf.d/abrt.conf', 'dest': '/etc/abrt/abrt.conf'}, 'ansible_loop_var': 'item', '_ansible_item_label': {'src': '/usr/share/abrt/conf.d/abrt.conf', 'dest': '/etc/abrt/abrt.conf'}}
2022-06-27 13:36:53,734+0530 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113 {'failed': True, 'msg': 'Source /usr/share/abrt/conf.d/plugins/CCpp.conf not found', 'invocation': {'module_args': {'remote_src': True, 'src': '/usr/share/abrt/conf.d/plugins/CCpp.conf', 'dest': '/etc/abrt/plugins/CCpp.conf', 'mode': 'preserve', 'backup': False, 'force': True, 'follow': False, 'unsafe_writes': False, '_original_basename': None, 'content': None, 'validate': None, 'directory_mode': None, 'local_follow': None, 'checksum': None, 'owner': None, 'group': None, 'seuser': None, 'serole': None, 'selevel': None, 'setype': None, 'attributes': None}}, '_ansible_no_log': False, 'changed': False, 'item': {'src': '/usr/share/abrt/conf.d/plugins/CCpp.conf', 'dest': '/etc/abrt/plugins/CCpp.conf'}, 'ansible_loop_var': 'item', '_ansible_item_label': {'src': '/usr/share/abrt/conf.d/plugins/CCpp.conf', 'dest': '/etc/abrt/plugins/CCpp.conf'}}
2022-06-27 13:36:53,835+0530 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113 {'failed': True, 'msg': 'Source /usr/share/abrt/conf.d/plugins/vmcore.conf not found', 'invocation': {'module_args': {'remote_src': True, 'src': '/usr/share/abrt/conf.d/plugins/vmcore.conf', 'dest': '/etc/abrt/plugins/vmcore.conf', 'mode': 'preserve', 'backup': False, 'force': True, 'follow': False, 'unsafe_writes': False, '_original_basename': None, 'content': None, 'validate': None, 'directory_mode': None, 'local_follow': None, 'checksum': None, 'owner': None, 'group': None, 'seuser': None, 'serole': None, 'selevel': None, 'setype': None, 'attributes': None}}, '_ansible_no_log': False, 'changed': False, 'item': {'src': '/usr/share/abrt/conf.d/plugins/vmcore.conf', 'dest': '/etc/abrt/plugins/vmcore.conf'}, 'ansible_loop_var': 'item', '_ansible_item_label': {'src': '/usr/share/abrt/conf.d/plugins/vmcore.conf', 'dest': '/etc/abrt/plugins/vmcore.conf'}}
2022-06-27 13:36:54,036+0530 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 PLAY RECAP [localhost] : ok: 71 changed: 20 unreachable: 0 skipped: 54 failed: 1
2022-06-27 13:36:54,137+0530 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:226 ansible-playbook rc: 2
2022-06-27 13:36:54,137+0530 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:233 ansible-playbook stdout:
2022-06-27 13:36:54,137+0530 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:236 ansible-playbook stderr:
2022-06-27 13:36:54,137+0530 DEBUG otopi.context context._executeMethod:145 method exception
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 132, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/misc.py", line 399, in _closeup
    self.initial_clean_up(bootstrap_vars, inventory_source)
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/misc.py", line 451, in initial_clean_up
    r = ah.run()
  File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/ansible_utils.py", line 240, in run
    raise RuntimeError(_('Failed executing ansible-playbook'))
RuntimeError: Failed executing ansible-playbook
2022-06-27 13:36:54,199+0530 ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Closing up': Failed executing ansible-playbook
~~~



Workaround:

Remove /etc/abrt/abrt-action-save-package-data.conf


Version-Release number of selected component (if applicable):
4.5.0 

How reproducible:
In customer environment.



Steps to Reproduce:
1. Install RHV-H 4.4.x 
2. Upgrade RHV-H 4.5.0
3. Try to deploy HE

Actual results:
HE deploy fail

Expected results:
HE deploy succeeds

Additional info:
If any logs are needed, let me know and I can provide them.
Comment 6 Nikolai Sednev 2022-08-02 09:56:37 UTC 
Is it RHVH specific?
Comment 7 Nikolai Sednev 2022-08-02 09:58:43 UTC 
What should be the reproduction steps?
Comment 20 Nikolai Sednev 2022-08-17 20:47:24 UTC 
1.Deployed 4.4 environment with rhvh-4.4.10.3-0.20220321.0+1 and RHEL8.5, with the engine 4.4.9.5-0.1.el8ev.
On RHVH:
abrt-2.10.9-21.el8.x86_64
ovirt-hosted-engine-ha-2.4.10-1.el8ev.noarch
ovirt-hosted-engine-setup-2.5.4-2.el8ev.noarch
Linux 4.18.0-348.20.1.el8_5.x86_64 #1 SMP Tue Mar 8 12:56:54 EST 2022 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux release 8.5

On RHEL:
abrt-2.10.9-21.el8.x86_64
ovirt-hosted-engine-ha-2.4.10-1.el8ev.noarch
ovirt-hosted-engine-setup-2.5.4-2.el8ev.noarch
Linux 4.18.0-348.23.1.el8_5.x86_64 #1 SMP Tue Apr 12 11:20:32 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux release 8.5 (Ootpa)

1.1. Upgraded the engine to latest bits of ovirt-engine-setup-4.4.10.8-548.g6b5767a.2.el8ev.noarch.

2.Upgraded the engine to latest 4.4 SP1 (ovirt-engine-setup-4.5.2.3-0.1.el8ev.noarch).
3.Upgraded RHV-H host to latest RHVH from 4.4 SP1 using Upgrade flow in webadmin.
3.3. 
redhat-virtualization-host-image-update-placeholder-4.5.2-1.el8ev.noarch
redhat-virtualization-host-image-update-4.5.2-202208170132_8.6.x86_64
ovirt-hosted-engine-setup-2.6.5-1.1.el8ev.noarch
ovirt-hosted-engine-ha-2.5.0-1.el8ev.noarch
Linux 4.18.0-372.23.1.el8_6.x86_64 #1 SMP Wed Aug 10 11:51:12 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux release 8.6

4.Upgraded RHEL-H host to latest RHEL from 4.4 SP1 using Upgrade flow in webadmin.
4.4.
ovirt-hosted-engine-setup-2.6.5-1.1.el8ev.noarch
ovirt-hosted-engine-ha-2.5.0-1.el8ev.noarch
Linux 4.18.0-372.23.1.el8_6.x86_64 #1 SMP Wed Aug 10 11:51:12 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux release 8.6 (Ootpa)
Comment 24 errata-xmlrpc 2022-09-08 11:29:03 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (RHV Engine and Host Common Packages update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:6394