Bug 1716594

Summary:

DCI-tempest environment creation failing(subnetpool not a supported extension)

Product:

Red Hat OpenStack

Reporter:

shaju <shajuvk>

Component:

openstack-tempest

Assignee:

Martin Kopec <mkopec>

Status:

CLOSED ERRATA

QA Contact:

Lukas Piwowarski <lpiwowar>

Severity:

high

Docs Contact:

Priority:

high

Version:

13.0 (Queens)

CC:

afazekas, amoralej, amuller, apevec, chkumar, fcharlie, lhh, ltoscano, mkopec, pamadio, sknauss, slinaber, udesale, whayutin

Target Milestone:

Keywords:

Triaged, ZStream

Target Release:

13.0 (Queens)

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

openstack-tempest-18.0.0-10.el7ost

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Clones:

1722147 (view as bug list)

Environment:

Last Closed:

2019-09-03 16:58:10 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

1722147, 1722154, 1737754, 1737757, 1737759

Attachments:

Description	Flags
tempest_output.log	none
tempest.log file from the latest run	none
tempest_output.log file from the latest run	none
test_fix.log	none

Description shaju 2019-06-03 17:37:39 UTC

Description of problem:
DCI tempest execution failed while creating tempest environment.

While listing subnetpool , tempest setup fails. Subnetpool extension is not a supported neutron plugin in contrail
Version-Release number of selected component (if applicable):


How reproducible:
Run the DCI tempest setup creation

Steps to Reproduce:
1.
2.
3.

Actual results:

019-05-29 17:46:30 | + /usr/bin/tempest cleanup --init-saved-state
2019-05-29 17:46:38 | Traceback (most recent call last):
2019-05-29 17:46:38 |   File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup.py", line 98, in take_action
2019-05-29 17:46:38 |     self.init(parsed_args)
2019-05-29 17:46:38 |   File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup.py", line 126, in init
2019-05-29 17:46:38 |     self._init_state()
2019-05-29 17:46:38 |   File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup.py", line 309, in _init_state
2019-05-29 17:46:38 |     svc.run()
2019-05-29 17:46:38 |   File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup_service.py", line 132, in run
2019-05-29 17:46:38 |     self.save_state()
2019-05-29 17:46:38 |   File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup_service.py", line 705, in save_state
2019-05-29 17:46:38 |     pools = self.list()
2019-05-29 17:46:38 |   File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup_service.py", line 680, in list
2019-05-29 17:46:38 |     pools = client.list_subnetpools(**self.tenant_filter)['subnetpools']
2019-05-29 17:46:38 |   File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/subnetpools_client.py", line 28, in list_subnetpools
2019-05-29 17:46:38 |     return self.list_resources(uri, **filters)
2019-05-29 17:46:38 |   File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/base.py", line 34, in list_resources
2019-05-29 17:46:38 |     resp, body = self.get(req_uri)
2019-05-29 17:46:38 |   File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 294, in get
2019-05-29 17:46:38 |     return self.request('GET', url, extra_headers, headers)
2019-05-29 17:46:38 |   File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 668, in request
2019-05-29 17:46:38 |     self._error_checker(resp, resp_body)
2019-05-29 17:46:38 |   File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 842, in _error_checker
2019-05-29 17:46:38 |     message=message)
2019-05-29 17:46:38 | NotImplemented: Got NotImplemented error
2019-05-29 17:46:38 | Details: 
2019-05-29 17:46:38 | Initializing saved state.
³!@)stdout'No tests in repository'
Expected results:


Additional info:

subnetpool not  a supported extension

Comment 1 shaju 2019-06-03 17:38:10 UTC

Created attachment 1576729 [details]
tempest_output.log

Comment 2 shaju 2019-06-03 18:13:37 UTC

with the latest run DCI failing way before tempest environment creation. It failed while pulling container image: 

Error
====
docker.errors.NotFound: 404 Client Error: Not Found ("{"message":"No such image: registry.distributed-ci.io/rhosp13/openstack-neutron-dhcp-agent:2019-05-31.1
===
non-zero return code
[{'_ansible_no_log': False,
  'attempts': 3,
  'changed': False,
  'msg': u'non-zero return code',
  'rc': 1,
  'stderr': u'Traceback (most recent call last):\n  File "/var/lib/dci-openstack-agent/.ansible/tmp/ansible-tmp-1559580390.38-73220090061182/fetch_images.py", line 141, in <module>\n    main()\n  File "/var/lib/dci-openstack-agent/.ansible/tmp/ansible-tmp-1559580390.38-73220090061182/fetch_images.py", line 134, in main\n    sync_image(image)\n  File "/var/lib/dci-openstack-agent/.ansible/tmp/ansible-tmp-1559580390.38-73220090061182/fetch_images.py", line 86, in sync_image\n    tag_image(image, tag)\n  File "/var/lib/dci-openstack-agent/.ansible/tmp/ansible-tmp-1559580390.38-73220090061182/fetch_images.py", line 44, in tag_image\n    docker_image_id = client.inspect_image(\'{origin_registry}/{project}/{name}:{tag}\'.format(**image))[\'Id\']\n  File "/usr/lib/python2.7/site-packages/docker/utils/decorators.py", line 21, in wrapped\n    return f(self, resource_id, *args, **kwargs)\n  File "/usr/lib/python2.7/site-packages/docker/api/image.py", line 141, in inspect_image\n    True\n  File "/usr/lib/python2.7/site-packages/docker/client.py", line 178, in _result\n    self._raise_for_status(response)\n  File "/usr/lib/python2.7/site-packages/docker/client.py", line 173, in _raise_for_status\n    raise errors.NotFound(e, response, explanation=explanation)\ndocker.errors.NotFound: 404 Client Error: Not Found ("{"message":"No such image: registry.distributed-ci.io/rhosp13/openstack-neutron-dhcp-agent:2019-05-31.1"}")\n',
  'stderr_lines': [u'Traceback (most recent call last):',
                   u'  File "/var/lib/dci-openstack-agent/.ansible/tmp/ansible-tmp-1559580390.38-73220090061182/fetch_images.py", line 141, in <module>',
                   u'    main()',
                   u'  File "/var/lib/dci-openstack-agent/.ansible/tmp/ansible-tmp-1559580390.38-73220090061182/fetch_images.py", line 134, in main',
                   u'    sync_image(image)',
                   u'  File "/var/lib/dci-openstack-agent/.ansible/tmp/ansible-tmp-1559580390.38-73220090061182/fetch_images.py", line 86, in sync_image',
                   u'    tag_image(image, tag)',
                   u'  File "/var/lib/dci-openstack-agent/.ansible/tmp/ansible-tmp-1559580390.38-73220090061182/fetch_images.py", line 44, in tag_image',
                   u"    docker_image_id = client.inspect_image('{origin_registry}/{project}/{name}:{tag}'.format(**image))['Id']",
                   u'  File "/usr/lib/python2.7/site-packages/docker/utils/decorators.py", line 21, in wrapped',
                   u'    return f(self, resource_id, *args, **kwargs)',
                   u'  File "/usr/lib/python2.7/site-packages/docker/api/image.py", line 141, in inspect_image',
                   u'    True',
                   u'  File "/usr/lib/python2.7/site-packages/docker/client.py", line 178, in _result',
                   u'    self._raise_for_status(response)',
                   u'  File "/usr/lib/python2.7/site-packages/docker/client.py", line 173, in _raise_for_status',
                   u'    raise errors.NotFound(e, response, explanation=explanation)',
                   u'docker.errors.NotFound: 404 Client Error: Not Found ("{"message":"No such image: registry.distributed-ci.io/rhosp13/openstack-neutron-dhcp-agent:2019-05-31.1"}")'],
  'stdout': u'login onto registry.distributed-ci.io\nImage openstack-aodh-api:2019-05-31.1 is already on the registry\nImage openstack-aodh-base:2019-05-31.1 is already on the registry\nImage openstack-aodh-evaluator:2019-05-31.1 is already on the registry\nImage openstack-aodh-listener:2019-05-31.1 is already on the registry\nImage openstack-aodh-notifier:2019-05-31.1 is already on the registry\nImage openstack-barbican-api:2019-05-31.1 is already on the registry\nImage openstack-barbican-base:2019-05-31.1 is already on the registry\nImage openstack-barbican-keystone-listener:2019-05-31.1 is already on the registry\nImage openstack-barbican-worker:2019-05-31.1 is already on the registry\nImage openstack-base:2019-05-31.1 is already on the registry\nImage openstack-ceilometer-base:2019-05-31.1 is already on the registry\nImage openstack-ceilometer-central:2019-05-31.1 is already on the registry\nImage openstack-ceilometer-compute:2019-05-31.1 is already on the registry\nImage openstack-ceilometer-ipmi:2019-05-31.1 is already on the registry\nImage openstack-ceilometer-notification:2019-05-31.1 is already on the registry\nImage openstack-cinder-api:2019-05-31.1 is already on the registry\nImage openstack-cinder-backup:2019-05-31.1 is already on the registry\nImage openstack-cinder-base:2019-05-31.1 is already on the registry\nImage openstack-cinder-scheduler:2019-05-31.1 is already on the registry\nImage openstack-cinder-volume:2019-05-31.1 is already on the registry\nImage openstack-collectd:2019-05-31.1 is already on the registry\nImage openstack-cron:2019-05-31.1 is already on the registry\nImage openstack-dependencies:2019-05-31.1 is already on the registry\nImage openstack-ec2-api:2019-05-31.1 is already on the registry\nImage openstack-etcd:2019-05-31.1 is already on the registry\nImage openstack-fluentd:2019-05-31.1 is already on the registry\nImage openstack-glance-api:2019-05-31.1 is already on the registry\nImage openstack-glance-base:2019-05-31.1 is already on the registry\nImage openstack-gnocchi-api:2019-05-31.1 is already on the registry\nImage openstack-gnocchi-base:2019-05-31.1 is already on the registry\nImage openstack-gnocchi-metricd:2019-05-31.1 is already on the registry\nImage openstack-gnocchi-statsd:2019-05-31.1 is already on the registry\nImage openstack-haproxy:2019-05-31.1 is already on the registry\nImage openstack-heat-all:2019-05-31.1 is already on the registry\nImage openstack-heat-api:2019-05-31.1 is already on the registry\nImage openstack-heat-api-cfn:2019-05-31.1 is already on the registry\nImage openstack-heat-base:2019-05-31.1 is already on the registry\nImage openstack-heat-engine:2019-05-31.1 is already on the registry\nImage openstack-horizon:2019-05-31.1 is already on the registry\nImage openstack-ironic-api:2019-05-31.1 is already on the registry\nImage openstack-ironic-base:2019-05-31.1 is already on the registry\nImage openstack-ironic-conductor:2019-05-31.1 is already on the registry\nImage openstack-ironic-inspector:2019-05-31.1 is already on the registry\nImage openstack-ironic-pxe:2019-05-31.1 is already on the registry\nImage openstack-iscsid:2019-05-31.1 is already on the registry\nImage openstack-keystone:2019-05-31.1 is already on the registry\nImage openstack-keystone-base:2019-05-31.1 is already on the registry\nImage openstack-kuryr-cni:2019-05-31.1 is already on the registry\nImage openstack-kuryr-controller:2019-05-31.1 is already on the registry\nImage openstack-manila-api:2019-05-31.1 is already on the registry\nImage openstack-manila-base:2019-05-31.1 is already on the registry\nImage openstack-manila-scheduler:2019-05-31.1 is already on the registry\nImage openstack-manila-share:2019-05-31.1 is already on the registry\nImage openstack-mariadb:2019-05-31.1 is already on the registry\nImage openstack-memcached:2019-05-31.1 is already on the registry\nImage openstack-mistral-api:2019-05-31.1 is already on the registry\nImage openstack-mistral-base:2019-05-31.1 is already on the registry\nImage openstack-mistral-engine:2019-05-31.1 is already on the registry\nImage openstack-mistral-event-engine:2019-05-31.1 is already on the registry\nImage openstack-mistral-executor:2019-05-31.1 is already on the registry\nImage openstack-multipathd:2019-05-31.1 is already on the registry\nImage openstack-neutron-base:2019-05-31.1 is already on the registry\ndocker pull registry.distributed-ci.io/rhosp13/openstack-neutron-dhcp-agent:2019-05-31.1\n',
  'stdout_lines': [u'login onto registry.distributed-ci.io',
                   u'Image openstack-aodh-api:2019-05-31.1 is already on the registry',
                   u'Image openstack-aodh-base:2019-05-31.1 is already on the registry',
                   u'Image openstack-aodh-evaluator:2019-05-31.1 is already on the registry',
                   u'Image openstack-aodh-listener:2019-05-31.1 is already on the registry',
                   u'Image openstack-aodh-notifier:2019-05-31.1 is already on the registry',
                   u'Image openstack-barbican-api:2019-05-31.1 is already on the registry',
                   u'Image openstack-barbican-base:2019-05-31.1 is already on the registry',
                   u'Image openstack-barbican-keystone-listener:2019-05-31.1 is already on the registry',
                   u'Image openstack-barbican-worker:2019-05-31.1 is already on the registry',
                   u'Image openstack-base:2019-05-31.1 is already on the registry',
                   u'Image openstack-ceilometer-base:2019-05-31.1 is already on the registry',
                   u'Image openstack-ceilometer-central:2019-05-31.1 is already on the registry',
                   u'Image openstack-ceilometer-compute:2019-05-31.1 is already on the registry',
                   u'Image openstack-ceilometer-ipmi:2019-05-31.1 is already on the registry',
                   u'Image openstack-ceilometer-notification:2019-05-31.1 is already on the registry',
                   u'Image openstack-cinder-api:2019-05-31.1 is already on the registry',
                   u'Image openstack-cinder-backup:2019-05-31.1 is already on the registry',
                   u'Image openstack-cinder-base:2019-05-31.1 is already on the registry',
                   u'Image openstack-cinder-scheduler:2019-05-31.1 is already on the registry',
                   u'Image openstack-cinder-volume:2019-05-31.1 is already on the registry',
                   u'Image openstack-collectd:2019-05-31.1 is already on the registry',
                   u'Image openstack-cron:2019-05-31.1 is already on the registry',
                   u'Image openstack-dependencies:2019-05-31.1 is already on the registry',
                   u'Image openstack-ec2-api:2019-05-31.1 is already on the registry',
                   u'Image openstack-etcd:2019-05-31.1 is already on the registry',
                   u'Image openstack-fluentd:2019-05-31.1 is already on the registry',
                   u'Image openstack-glance-api:2019-05-31.1 is already on the registry',
                   u'Image openstack-glance-base:2019-05-31.1 is already on the registry',
                   u'Image openstack-gnocchi-api:2019-05-31.1 is already on the registry',
                   u'Image openstack-gnocchi-base:2019-05-31.1 is already on the registry',
                   u'Image openstack-gnocchi-metricd:2019-05-31.1 is already on the registry',
                   u'Image openstack-gnocchi-statsd:2019-05-31.1 is already on the registry',
                   u'Image openstack-haproxy:2019-05-31.1 is already on the registry',
                   u'Image openstack-heat-all:2019-05-31.1 is already on the registry',
                   u'Image openstack-heat-api:2019-05-31.1 is already on the registry',
                   u'Image openstack-heat-api-cfn:2019-05-31.1 is already on the registry',
                   u'Image openstack-heat-base:2019-05-31.1 is already on the registry',
                   u'Image openstack-heat-engine:2019-05-31.1 is already on the registry',
                   u'Image openstack-horizon:2019-05-31.1 is already on the registry',
                   u'Image openstack-ironic-api:2019-05-31.1 is already on the registry',
                   u'Image openstack-ironic-base:2019-05-31.1 is already on the registry',
                   u'Image openstack-ironic-conductor:2019-05-31.1 is already on the registry',
                   u'Image openstack-ironic-inspector:2019-05-31.1 is already on the registry',
                   u'Image openstack-ironic-pxe:2019-05-31.1 is already on the registry',
                   u'Image openstack-iscsid:2019-05-31.1 is already on the registry',
                   u'Image openstack-keystone:2019-05-31.1 is already on the registry',
                   u'Image openstack-keystone-base:2019-05-31.1 is already on the registry',
                   u'Image openstack-kuryr-cni:2019-05-31.1 is already on the registry',
                   u'Image openstack-kuryr-controller:2019-05-31.1 is already on the registry',
                   u'Image openstack-manila-api:2019-05-31.1 is already on the registry',
                   u'Image openstack-manila-base:2019-05-31.1 is already on the registry',
                   u'Image openstack-manila-scheduler:2019-05-31.1 is already on the registry',
                   u'Image openstack-manila-share:2019-05-31.1 is already on the registry',
                   u'Image openstack-mariadb:2019-05-31.1 is already on the registry',
                   u'Image openstack-memcached:2019-05-31.1 is already on the registry',
                   u'Image openstack-mistral-api:2019-05-31.1 is already on the registry',
                   u'Image openstack-mistral-base:2019-05-31.1 is already on the registry',
                   u'Image openstack-mistral-engine:2019-05-31.1 is already on the registry',
                   u'Image openstack-mistral-event-engine:2019-05-31.1 is already on the registry',
                   u'Image openstack-mistral-executor:2019-05-31.1 is already on the registry',
                   u'Image openstack-multipathd:2019-05-31.1 is already on the registry',
                   u'Image openstack-neutron-base:2019-05-31.1 is already on the registry',
                   u'docker pull registry.distributed-ci.io/rhosp13/openstack-neutron-dhcp-agent:2019-05-31.1']}]

Comment 3 François Charlier 2019-06-04 14:24:58 UTC

As explained through direct email, the issue with the missing container image is fixed,

can you restart the agent again ?

Comment 4 shaju 2019-06-05 17:54:09 UTC

Thanks for the help. Latest DCI didn't hit the docker image issue.

But we still have the tempest  environment issue Subnetpool. Could you please fix this too

Comment 5 shaju 2019-06-05 18:22:39 UTC

Noticed below error in tempest_output.log

======

2019-06-05 13:34:48 | 2019-06-05 13:34:48.521 31836 DEBUG config_tempest.constants [-] Setting [volume] max_microversion = 3.50 set /usr/lib/python2.7/site-packages/config_tempest/tempest_conf.py:103
2019-06-05 13:34:48 | 2019-06-05 13:34:48.834 31836 INFO config_tempest.constants [-] Service metering has no endpoints
2019-06-05 13:34:48 | 2019-06-05 13:34:48.835 31836 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /info: BadStatusLine: ''
2019-06-05 13:34:48 | 2019-06-05 13:34:48.837 31836 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /info: BadStatusLine: ''
2019-06-05 13:34:48 | 2019-06-05 13:34:48.838 31836 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /info: BadStatusLine: ''
2019-06-05 13:34:48 | Traceback (most recent call last):
2019-06-05 13:34:48 |   File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
2019-06-05 13:34:48 |     msg = self.format(record)
2019-06-05 13:34:48 |   File "/usr/lib/python2.7/site-packages/oslo_log/handlers.py", line 161, in format
2019-06-05 13:34:48 |     return logging.StreamHandler.format(self, record)
2019-06-05 13:34:48 |   File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
2019-06-05 13:34:48 |     return fmt.format(record)
2019-06-05 13:34:48 |   File "/usr/lib/python2.7/site-packages/oslo_log/formatters.py", line 501, in format
2019-06-05 13:34:48 |     return logging.Formatter.format(self, record)
2019-06-05 13:34:48 |   File "/usr/lib64/python2.7/logging/__init__.py", line 464, in format
2019-06-05 13:34:48 |     record.message = record.getMessage()
2019-06-05 13:34:48 |   File "/usr/lib64/python2.7/logging/__init__.py", line 328, in getMessage
2019-06-05 13:34:48 |     msg = msg % self.args
2019-06-05 13:34:48 | TypeError: not all arguments converted during string formatting
2019-06-05 13:34:48 | Logged from file base.py, line 60
2019-06-05 13:34:49 | 2019-06-05 13:34:49.113 31836 DEBUG config_tempest.constants [-] Option '[compute-feature-enabled] console_output = true' was defined by user, NOT overwriting into value 'True' set /usr/lib/python2.7/site-packages/config_tempest/tempest_conf.py:

Comment 6 François Charlier 2019-06-06 08:44:13 UTC

Created attachment 1577800 [details]
tempest.log file from the latest run

Comment 7 François Charlier 2019-06-06 08:44:49 UTC

Created attachment 1577801 [details]
tempest_output.log file from the latest run

Comment 8 François Charlier 2019-06-06 09:04:30 UTC

I added the latest logfiles to this BZ.

Other logfiles for the overcloud servers/services are available from this page: https://www.distributed-ci.io/jobs/588fd57f-3d59-4e82-a291-e89122baf15a/files

Comment 9 François Charlier 2019-06-06 09:05:56 UTC

The latest tempest logs have been added to the BZ, other logfiles for the overcloud servers/services are available from the following page: https://www.distributed-ci.io/jobs/588fd57f-3d59-4e82-a291-e89122baf15a/files

Comment 10 François Charlier 2019-06-11 15:25:06 UTC

Looking at tempest history, this issue has been introduced on Feb 19th 2019 in https://github.com/openstack/tempest/commit/e681998023d9714da346d117e18ab673b5657f71

This is a blocking issue for Juniper as they can't run tempest at all anymore because this is not implemented in their Neutron plugin.

Comment 11 Assaf Muller 2019-06-11 16:50:06 UTC

Is a Juniper Engineering going to send a patch to the Neutron Tempest plugin to make the subnetpool cleanup conditional on the subnetpool config option in tempest.conf? Or do they expect RH to send the patch? (if so why?)

Comment 13 wes hayutin 2019-06-27 17:13:52 UTC

How can we get this into a build for dci?

Comment 16 Martin Kopec 2019-07-04 19:56:07 UTC

*** Bug 1726936 has been marked as a duplicate of this bug. ***

Comment 23 Martin Kopec 2019-08-06 07:35:01 UTC

The issue is fixed only partially. tempest cleanup doesn't fail on NotImplemented error anymore and continues cleaning resources as expected, however, the return code of 'tempest cleanup --init-saved-state' is not appropriate and causes issues from automation perspective. This return code issue is tracked separately here https://bugzilla.redhat.com/show_bug.cgi?id=1737754

Comment 33 Lukas Piwowarski 2019-08-21 14:57:20 UTC

Created attachment 1606582 [details]
test_fix.log

Comment 34 Lukas Piwowarski 2019-08-21 14:58:41 UTC

Tested with:
  - package with the bug (openstack-tempest-18.0.0-8.el7ost)
  - package with the fix (openstack-tempest-18.0.0-10.el7ost)

The bug was not reproduced when tested with the package with the fix as shown in the attached log file (test_fix.log).

Comment 36 errata-xmlrpc 2019-09-03 16:58:10 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2623