Bug 1716350

Summary: [osp15] A stock osp15 deployment is full of aodh/gnocchi exceptions
Product: Red Hat OpenStack Reporter: Michele Baldessari <michele>
Component: openstack-aodhAssignee: Martin Magr <mmagr>
Status: CLOSED ERRATA QA Contact: Nataf Sharabi <nsharabi>
Severity: high Docs Contact:
Priority: high    
Version: 15.0 (Stein)CC: apevec, jpichon, jschluet, lhh, mmagr, mrunge, pkilambi, vvoronko
Target Milestone: rcKeywords: Triaged
Target Release: 15.0 (Stein)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-aodh-8.0.1-0.20190829102418.1b1b8e2.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-09-21 11:22:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Michele Baldessari 2019-06-03 09:49:31 UTC 
Description of problem:
While looking at https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP15/job/phase2-15_director-rhel-8.0-virthost-3cont_2comp-ipv4-geneve-ceph-external/lastCompletedBuild/

I noticed a bunch of aodh and gnocchi exceptions:
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:37.158 20 WARNING aodh.coordination [-] Cannot extract tasks because agent failed to join group properly. Rejoining group.                                                                            controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:38.904 20 WARNING aodh.coordination [-] Cannot extract tasks because agent failed to join group properly. Rejoining group.
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:40.150 20 WARNING aodh.coordination [-] Cannot extract tasks because agent failed to join group properly. Rejoining group.
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:41.744 20 WARNING aodh.coordination [-] Cannot extract tasks because agent failed to join group properly. Rejoining group.
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.719 20 WARNING aodh.coordination [-] Cannot extract tasks because agent failed to join group properly. Rejoining group.
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator [-] alarm evaluation cycle failed: aodh.coordination.MemberNotInGroupError: Group ID: alarm_evaluator, Members: {b'1dbcfdf7-ab3d-446a-8be1-a5f461687cd6', b'c8bc9aeb-5e56-4bec-892b-5ff83617618a', b'e2baa83f-6e20-4ab3-ab42-b1c227210be2'}, Me: e2baa83f-6e20-4ab3-ab42-b1c227210be2: Current agent is not part of group and cannot take tasks
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator Traceback (most recent call last):
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator   File "/usr/lib/python3.6/site-packages/aodh/evaluator/__init__.py", line 249, in _evaluate_assigned_alarms
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator     alarms = self._assigned_alarms()
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator   File "/usr/lib/python3.6/site-packages/aodh/evaluator/__init__.py", line 278, in _assigned_alarms
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator     self.PARTITIONING_GROUP_NAME, all_alarm_ids) 
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator   File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 292, in wrapped_f
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator     return self.call(f, *args, **kw)
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator   File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 358, in call
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator     do = self.iter(retry_state=retry_state)
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator   File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 331, in iter
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator     raise retry_exc.reraise()
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator   File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 167, in reraise
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator     raise self.last_attempt.result()
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator   File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator     return self.__get_result()
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator   File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator     raise self._exception
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator   File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 361, in call
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator     result = fn(*args, **kwargs)
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator   File "/usr/lib/python3.6/site-packages/aodh/coordination.py", line 234, in extract_my_subset
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator     raise MemberNotInGroupError(group_id, members, self._my_id)
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator aodh.coordination.MemberNotInGroupError: Group ID: alarm_evaluator, Members: {b'1dbcfdf7-ab3d-446a-8be1-a5f461687cd6', b'c8bc9aeb-5e56-4bec-892b-5ff83617618a', b'e2baa83f-6e20-4ab3-ab42-b1c227210be2'}, Me: e2baa83f-6e20-4ab3-ab42-b1c227210be2: Current agent is not part of group and cannot take tasks
controller-0/var/log/containers/aodh/aodh-evaluator.log:2019-05-31 21:39:43.724 20 ERROR aodh.evaluator 

I assume the gnocchi failures are related as well:
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 21:57:06.429 23 ERROR ceilometer.publisher.gnocchi [-] Fail to update the resource {'type': 'instance', 'id': '0e1ef903-fca6-45b0-8e17-65c45a04b4f5'}: gnocchiclient.exceptions.BadRequest: Value 2019-05-31 21:57:05.334071+00:00 for attribute ended_at on resource type instance is invalid (HTTP 400)                                                                                                                                                                        
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 21:57:06.429 23 ERROR ceilometer.publisher.gnocchi Traceback (most recent call last):
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 21:57:06.429 23 ERROR ceilometer.publisher.gnocchi   File "/usr/lib/python3.6/site-packages/ceilometer/publisher/gnocchi.py", line 527, in _set_ended_at
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 21:57:06.429 23 ERROR ceilometer.publisher.gnocchi     {'ended_at': ended_at})
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 21:57:06.429 23 ERROR ceilometer.publisher.gnocchi   File "/usr/lib/python3.6/site-packages/gnocchiclient/v1/resource.py", line 113, in update
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 21:57:06.429 23 ERROR ceilometer.publisher.gnocchi     data=ujson.dumps(resource)).json()
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 21:57:06.429 23 ERROR ceilometer.publisher.gnocchi   File "/usr/lib/python3.6/site-packages/gnocchiclient/v1/base.py", line 49, in _patch
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 21:57:06.429 23 ERROR ceilometer.publisher.gnocchi     return self.client.api.patch(*args, **kwargs)
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 21:57:06.429 23 ERROR ceilometer.publisher.gnocchi   File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 387, in patch 
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 21:57:06.429 23 ERROR ceilometer.publisher.gnocchi     return self.request(url, 'PATCH', **kwargs)
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 21:57:06.429 23 ERROR ceilometer.publisher.gnocchi   File "/usr/lib/python3.6/site-packages/gnocchiclient/client.py", line 52, in request
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 21:57:06.429 23 ERROR ceilometer.publisher.gnocchi     raise exceptions.from_response(resp, method)
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 21:57:06.429 23 ERROR ceilometer.publisher.gnocchi gnocchiclient.exceptions.BadRequest: Value 2019-05-31 21:57:05.334071+00:00 for attribute ended_at on resource type instance is invalid (HTTP 400)
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 21:57:06.429 23 ERROR ceilometer.publisher.gnocchi 
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 22:07:55.286 23 ERROR ceilometer.publisher.gnocchi [-] Fail to update the resource {'type': 'instance', 'id': '8c7f669e-c020-4276-b2ed-f4dfecf3928e'}: gnocchiclient.exceptions.BadRequest: Value 2019-05-31 22:07:55.075419+00:00 for attribute ended_at on resource type instance is invalid (HTTP 400)
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 22:07:55.286 23 ERROR ceilometer.publisher.gnocchi Traceback (most recent call last):
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 22:07:55.286 23 ERROR ceilometer.publisher.gnocchi   File "/usr/lib/python3.6/site-packages/ceilometer/publisher/gnocchi.py", line 527, in _set_ended_at
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 22:07:55.286 23 ERROR ceilometer.publisher.gnocchi     {'ended_at': ended_at})
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 22:07:55.286 23 ERROR ceilometer.publisher.gnocchi   File "/usr/lib/python3.6/site-packages/gnocchiclient/v1/resource.py", line 113, in update
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 22:07:55.286 23 ERROR ceilometer.publisher.gnocchi     data=ujson.dumps(resource)).json()
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 22:07:55.286 23 ERROR ceilometer.publisher.gnocchi   File "/usr/lib/python3.6/site-packages/gnocchiclient/v1/base.py", line 49, in _patch
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 22:07:55.286 23 ERROR ceilometer.publisher.gnocchi     return self.client.api.patch(*args, **kwargs)
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 22:07:55.286 23 ERROR ceilometer.publisher.gnocchi   File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 387, in patch 
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 22:07:55.286 23 ERROR ceilometer.publisher.gnocchi     return self.request(url, 'PATCH', **kwargs)
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 22:07:55.286 23 ERROR ceilometer.publisher.gnocchi   File "/usr/lib/python3.6/site-packages/gnocchiclient/client.py", line 52, in request
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 22:07:55.286 23 ERROR ceilometer.publisher.gnocchi     raise exceptions.from_response(resp, method)
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 22:07:55.286 23 ERROR ceilometer.publisher.gnocchi gnocchiclient.exceptions.BadRequest: Value 2019-05-31 22:07:55.075419+00:00 for attribute ended_at on resource type instance is invalid (HTTP 400)
controller-0/var/log/containers/ceilometer/agent-notification.log:2019-05-31 22:07:55.286 23 ERROR ceilometer.publisher.gnocchi 
controller-0/var/log/containers/ceilometer/central.log:2019-05-31 21:38:42.062 8 DEBUG cotyledon.oslo_config_glue [-] logging_exception_prefix       = %(asctime)s.%(msecs)03d %(process)d ERROR %(name)s %(instance)s log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2571
controller-0/var/log/containers/ceilometer/central.log:2019-05-31 21:38:42.189 22 ERROR ceilometer.hardware.pollsters.generic [-] Error loading meter definition: Missing field unit: ceilometer.declarative.MeterDefinitionException: MeterDefinitionException {'name': 'hardware.cpu.user', 'snmp_inspector': {'matching_type': 'type_exact', 'oid': '1.3.6.1.4.1.2021.11.50.0', 'type': 'int'}}: Missing field unit
controller-0/var/log/containers/ceilometer/central.log:2019-05-31 21:38:42.190 22 ERROR ceilometer.hardware.pollsters.generic [-] Error loading meter definition: Missing field unit: ceilometer.declarative.MeterDefinitionException: MeterDefinitionException {'name': 'hardware.cpu.nice', 'snmp_inspector': {'matching_type': 'type_exact', 'oid': '1.3.6.1.4.1.2021.11.51.0', 'type': 'int'}}: Missing field unit
controller-0/var/log/containers/ceilometer/central.log:2019-05-31 21:38:42.191 22 ERROR ceilometer.hardware.pollsters.generic [-] Error loading meter definition: Missing field unit: ceilometer.declarative.MeterDefinitionException: MeterDefinitionException {'name': 'hardware.cpu.system', 'snmp_inspector': {'matching_type': 'type_exact', 'oid': '1.3.6.1.4.1.2021.11.52.0', 'type': 'int'}}: Missing field unit
Comment 10 Martin Magr 2019-06-17 14:37:45 UTC 
*** Bug 1715114 has been marked as a duplicate of this bug. ***
Comment 15 Nataf Sharabi 2019-08-28 09:58:14 UTC 
Hi,

I've installed  core_puddle=RHOS_TRUNK-15.0-RHEL-8-20190823.n.1

with topology-nodes undercloud:1,controller:3,compute:2,freeipa:1 

I can still see the errors:

019-08-28 00:01:29.278 21 ERROR aodh.evaluator [-] alarm evaluation cycle failed: aodh.coordination.MemberNotInGroupError: Group ID: alarm_evaluator, Members: {b'b9e8ecbc-39ac-4baf-9310-d726c4a18a55', b'aeb60e48-12fc-44f2-8792-fe73941e9618', b'000cae04-b67d-4213-9fc4-9dc08e52d723'}, Me: 000cae04-b67d-4213-9fc4-9dc08e52d723: Current agent is not part of group and cannot take tasks
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator Traceback (most recent call last):
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator   File "/usr/lib/python3.6/site-packages/aodh/evaluator/__init__.py", line 249, in _evaluate_assigned_alarms
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator     alarms = self._assigned_alarms()
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator   File "/usr/lib/python3.6/site-packages/aodh/evaluator/__init__.py", line 278, in _assigned_alarms
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator     self.PARTITIONING_GROUP_NAME, all_alarm_ids)
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator   File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 292, in wrapped_f
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator     return self.call(f, *args, **kw)
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator   File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 358, in call
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator     do = self.iter(retry_state=retry_state)
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator   File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 331, in iter
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator     raise retry_exc.reraise()
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator   File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 167, in reraise
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator     raise self.last_attempt.result()
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator   File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator     return self.__get_result()
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator   File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator     raise self._exception
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator   File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 361, in call
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator     result = fn(*args, **kwargs)
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator   File "/usr/lib/python3.6/site-packages/aodh/coordination.py", line 234, in extract_my_subset
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator     raise MemberNotInGroupError(group_id, members, self._my_id)
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator aodh.coordination.MemberNotInGroupError: Group ID: alarm_evaluator, Members: {b'b9e8ecbc-39ac-4baf-9310-d726c4a18a55', b'aeb60e48-12fc-44f2-8792-fe73941e9618', b'000cae04-b67d-4213-9fc4-9dc08e52d723'}, Me: 000cae04-b67d-4213-9fc4-9dc08e52d723: Current agent is not part of group and cannot take tasks
2019-08-28 00:01:29.278 21 ERROR aodh.evaluator 
2019-08-28 00:02:24.595 21 WARNING oslo_db.sqlalchemy.utils [-] Unique keys not in sort_keys. The sorting order may be unstable.
2019-08-28 00:02:24.617 21 WARNING aodh.coordination [-] Cannot extract tasks because agent failed to join group properly. Rejoining group.
2019-08-28 00:02:26.489 21 WARNING aodh.coordination [-] Cannot extract tasks because agent failed to join group properly. Rejoining group.
2019-08-28 00:02:28.001 21 WARNING aodh.coordination [-] Cannot extract tasks because agent failed to join group properly. Rejoining group.
2019-08-28 00:02:29.608 21 WARNING aodh.coordination [-] Cannot extract tasks because agent failed to join group properly. Rejoining group.
2019-08-28 00:02:31.574 21 WARNING aodh.coordination [-] Cannot extract tasks because agent failed to join group properly. Rejoining group.
2019-08-28 00:02:31.587 21 ERROR aodh.evaluator [-] alarm evaluation cycle failed: aodh.coordination.MemberNotInGroupError: Group ID: alarm_evaluator, Members: {b'b9e8ecbc-39ac-4baf-9310-d726c4a18a55', b'aeb60e48-12fc-44f2-8792-fe73941e9618', b'000cae04-b67d-4213-9fc4-9dc08e52d723'}, Me: 000cae04-b67d-4213-9fc4-9dc08e52d723: Current agent is not part of group and cannot take tasks
2019-08-28 00:02:31.587 21 ERROR aodh.evaluator Traceback (most recent call last):
2019-08-28 00:02:31.587 21 ERROR aodh.evaluator   File "/usr/lib/python3.6/site-packages/aodh/evaluator/__init__.py", line 249, in _evaluate_assigned_alarms
2019-08-28 00:02:31.587 21 ERROR aodh.evaluator     alarms = self._assigned_alarms()
2019-08-28 00:02:31.587 21 ERROR aodh.evaluator   File "/usr/lib/python3.6/site-packages/aodh/evaluator/__init__.py", line 278, in _assigned_alarms
2019-08-28 00:02:31.587 21 ERROR aodh.evaluator     self.PARTITIONING_GROUP_NAME, all_alarm_ids)
2019-08-28 00:02:31.587 21 ERROR aodh.evaluator   File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 292, in wrapped_f
2019-08-28 00:02:31.587 21 ERROR aodh.evaluator     return self.call(f, *args, **kw)
2019-08-28 00:02:31.587 21 ERROR aodh.evaluator   File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 358, in call
Comment 16 Matthias Runge 2019-08-29 09:26:21 UTC 
rpm -q
Comment 17 Matthias Runge 2019-08-29 09:38:49 UTC 
To complete info here:

podman exec -it aodh_evaluator bash
openstack-aodh-evaluator-8.0.1-0.20190705170401.1b1b8e2.el8ost.noarch

This package does NOT contain the fix from Vinay. Apparently, the package was not rebuilt after the fix.
Comment 18 Matthias Runge 2019-08-29 10:14:41 UTC 
Scratch the last sentence please. 

https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=925978 aka openstack-aodh-8.0.1-0.20190705160402.1b1b8e2.el8ost
has the fix, the later version openstack-aodh-evaluator-8.0.1-0.20190705170401.1b1b8e2.el8ost.noarch has not.
Comment 22 errata-xmlrpc 2019-09-21 11:22:59 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811