Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1821144

Summary: rabbitmq plugin tries docker exec for stopped containers
Product: Red Hat Enterprise Linux 7 Reporter: Takashi Kajinami <tkajinam>
Component: sosAssignee: Jan Jansky <jjansky>
Status: CLOSED ERRATA QA Contact: Miroslav Hradílek <mhradile>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.6CC: agk, bmr, fkrska, jjansky, khoes, mhradile, plambri, pmoravec, sbradley
Target Milestone: rcKeywords: OtherQA
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: sos-3.9-2.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-09-29 20:55:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
sosreport generated by sos-3.7-10.el7_7.noarch
none
sosreport generated by sos-3.9-2.el7.noarch none

Description Takashi Kajinami 2020-04-06 07:17:18 UTC 
Description of problem:

When running sosreport on osp13 deployment on rhel7, it is observed that rabbitmq plugin tried to invoke "docker exec" for stopped docker containers, and the following errors are recorded in messages

dockerd-current: time="2020-04-01T12:34:56.789012345+00:00" level=error msg="Error setting up exec command in container rabbitmq_init_bundle: Container <id> is not running"

This error is caused by the behaviour in rabbitmq plugin, which makes the plugin to try execute docker exec command for all containers whose name start with "rabbitmq"
regardless of status of these containers
 https://github.com/sosreport/sos/blob/3.6/sos/plugins/rabbitmq.py#L37-L42


Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. run sosreport in osp13 deployment

Actual results:
sosreport tries to invoke "docker exec" for stopped containers, and error from dockerd-current is logged in messages

Expected results:
sosreport never tries to invoke "docker exec" for stopped containers, and error from dockerd-current is not logged in messages

Additional info:
Comment 5 Jan Jansky 2020-06-01 14:18:34 UTC 
Hello,
thanks for the availability to test the fix. Please use below repository / package:


A yum repository for the build of sos-3.9-2.el7 (task 28860092) is available at:

http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.9/2.el7/

You can install the rpms locally by putting this .repo file in your /etc/yum.repos.d/ directory:

http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.9/2.el7/sos-3.9-2.el7.repo

RPMs and build logs can be found in the following locations:
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.9/2.el7/noarch/

The full list of available rpms is:
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.9/2.el7/noarch/sos-3.9-2.el7.src.rpm
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.9/2.el7/noarch/sos-3.9-2.el7.noarch.rpm

The repository will be available for the next 60 days. Scratch build output will be deleted
earlier, based on the Brew scratch build retention policy.
Comment 6 Takashi Kajinami 2020-06-05 02:29:49 UTC 
I confirmed that after updating sos package, I don't see the error from dockerd-current
which was caused by exec command against stopped containers

(1) Before updating sos
~~~
[heat-admin@controller-0 ~]$ rpm -qa | grep sos
sos-3.7-10.el7_7.noarch
[heat-admin@controller-0 ~]$ date
Fri Jun  5 02:23:40 UTC 2020
[heat-admin@controller-0 ~]$ sudo sosreport -o rabbitmq

sosreport (version 3.7)

WARNING: unable to set option for disabled or non-existing plugin (process)

This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/var/tmp/sos.0sVf45 and may be provided to a Red Hat support
representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

  https://access.redhat.com/support/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Please enter the case id that you are generating this report for []: 

 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Starting 1/1   rabbitmq        [Running: rabbitmq]                                      
  Finished running plugins                                                               
Creating compressed archive...

Your sosreport has been generated and saved in:
  /var/tmp/sosreport-controller-0-2020-06-05-aukaxgm.tar.xz

The checksum is: e726fd20c730cb48cd586a73e14855bf

Please send this file to your support representative.

[heat-admin@controller-0 ~]$ date
Fri Jun  5 02:24:10 UTC 2020
[heat-admin@controller-0 ~]$ sudo grep "Jun  5 02:2[34]" /var/log/messages | grep dockerd-current | grep "level=err"
Jun  5 02:23:52 controller-0 dockerd-current: time="2020-06-05T02:23:52.630866622Z" level=error msg="Error setting up exec command in container rabbitmq_init_bundle: Container 325dc17b34004758990be2bc2a6a5c8a647a9458cd404a106fe63d8fc5395d1d is not running"
Jun  5 02:23:52 controller-0 dockerd-current: time="2020-06-05T02:23:52.630889405Z" level=error msg="Handler for POST /v1.26/containers/rabbitmq_init_bundle/exec returned error: Container 325dc17b34004758990be2bc2a6a5c8a647a9458cd404a106fe63d8fc5395d1d is not running"
Jun  5 02:23:52 controller-0 dockerd-current: time="2020-06-05T02:23:52.631175639Z" level=error msg="Handler for POST /v1.26/containers/rabbitmq_init_bundle/exec returned error: Container 325dc17b34004758990be2bc2a6a5c8a647a9458cd404a106fe63d8fc5395d1d is not running"
Jun  5 02:23:57 controller-0 dockerd-current: time="2020-06-05T02:23:57.631137241Z" level=error msg="Error setting up exec command in container rabbitmq_restart_bundle: Container 84537fffa1f2a553b8cdb8e78fb0de63d0de54bdb778a340c29f3eedbc72f906 is not running"
Jun  5 02:23:57 controller-0 dockerd-current: time="2020-06-05T02:23:57.631156820Z" level=error msg="Handler for POST /v1.26/containers/rabbitmq_restart_bundle/exec returned error: Container 84537fffa1f2a553b8cdb8e78fb0de63d0de54bdb778a340c29f3eedbc72f906 is not running"
Jun  5 02:23:57 controller-0 dockerd-current: time="2020-06-05T02:23:57.631409552Z" level=error msg="Handler for POST /v1.26/containers/rabbitmq_restart_bundle/exec returned error: Container 84537fffa1f2a553b8cdb8e78fb0de63d0de54bdb778a340c29f3eedbc72f906 is not running"
Jun  5 02:23:57 controller-0 dockerd-current: time="2020-06-05T02:23:57.715895565Z" level=error msg="Error setting up exec command in container rabbitmq_image_tag: Container ad4f9b276c47f54d170f077d9a1b2c242db2c97795c83615b8575774f4ff1d39 is not running"
Jun  5 02:23:57 controller-0 dockerd-current: time="2020-06-05T02:23:57.715916999Z" level=error msg="Handler for POST /v1.26/containers/rabbitmq_image_tag/exec returned error: Container ad4f9b276c47f54d170f077d9a1b2c242db2c97795c83615b8575774f4ff1d39 is not running"
Jun  5 02:23:57 controller-0 dockerd-current: time="2020-06-05T02:23:57.716168290Z" level=error msg="Handler for POST /v1.26/containers/rabbitmq_image_tag/exec returned error: Container ad4f9b276c47f54d170f077d9a1b2c242db2c97795c83615b8575774f4ff1d39 is not running"
Jun  5 02:23:57 controller-0 dockerd-current: time="2020-06-05T02:23:57.800577213Z" level=error msg="Error setting up exec command in container rabbitmq_bootstrap: Container a3fbefb576f96f05d49501be6c617d631cab3773d5153563fcf60edda82a7e1b is not running"
Jun  5 02:23:57 controller-0 dockerd-current: time="2020-06-05T02:23:57.800597495Z" level=error msg="Handler for POST /v1.26/containers/rabbitmq_bootstrap/exec returned error: Container a3fbefb576f96f05d49501be6c617d631cab3773d5153563fcf60edda82a7e1b is not running"
Jun  5 02:23:57 controller-0 dockerd-current: time="2020-06-05T02:23:57.800840638Z" level=error msg="Handler for POST /v1.26/containers/rabbitmq_bootstrap/exec returned error: Container a3fbefb576f96f05d49501be6c617d631cab3773d5153563fcf60edda82a7e1b is not running"
~~~

(2) After updating sos
~~~
[heat-admin@controller-0 ~]$ sudo rpm -Uvh sos-3.9-2.el7.noarch.rpm 
Preparing...                          ################################# [100%]
Updating / installing...
   1:sos-3.9-2.el7                    ################################# [ 50%]
Cleaning up / removing...
   2:sos-3.7-10.el7_7                 ################################# [100%]
[heat-admin@controller-0 ~]$ rpm -qa | grep sos
sos-3.9-2.el7.noarch
[heat-admin@controller-0 ~]$ date
Fri Jun  5 02:26:17 UTC 2020
[heat-admin@controller-0 ~]$ sudo sosreport -o rabbitmq

sosreport (version 3.9)

WARNING: unable to set option for disabled or non-existing plugin (process)
WARNING: unable to set option for disabled or non-existing plugin (process)
WARNING: unable to set option for disabled or non-existing plugin (networking)
WARNING: unable to set option for disabled or non-existing plugin (networking)


This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/var/tmp/sos.kUrC7N and may be provided to a Red Hat support
representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

  https://access.redhat.com/support/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Please enter the case id that you are generating this report for []: 

 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Starting 1/1   rabbitmq        [Running: rabbitmq]                                      
  Finished running plugins                                                               
Creating compressed archive...

Your sosreport has been generated and saved in:
  /var/tmp/sosreport-controller-0-2020-06-05-fptrhjk.tar.xz

 Size	82.99KiB
 Owner	root
 md5	eba95d1953f0e96d04feb27f1321a24e

Please send this file to your support representative.

[heat-admin@controller-0 ~]$ date
Fri Jun  5 02:26:51 UTC 2020
[heat-admin@controller-0 ~]$ sudo grep "Jun  5 02:26" /var/log/messages | grep dockerd-current | grep "level=err"
[heat-admin@controller-0 ~]$ 
~~~
Comment 10 Takashi Kajinami 2020-06-08 23:13:00 UTC 
Created attachment 1696218 [details]
sosreport generated by sos-3.7-10.el7_7.noarch
Comment 11 Takashi Kajinami 2020-06-08 23:14:15 UTC 
Created attachment 1696219 [details]
sosreport generated by sos-3.9-2.el7.noarch
Comment 12 Takashi Kajinami 2020-06-08 23:21:52 UTC 
I've attached sosreport generated during my testing.

I confirmed that sosreport includes expected results (though unfortunately docker logs shows empty),
and there is no error recorded in sos.log .
~~~
$ ls sosreport-controller-0-2020-06-05-*/sos_commands/rabbitmq/
sosreport-controller-0-2020-06-05-aukaxgm/sos_commands/rabbitmq/:
docker_exec_-t_rabbitmq_bootstrap_rabbitmqctl_report        docker_logs_rabbitmq_bootstrap
docker_exec_-t_rabbitmq-bundle-docker-0_rabbitmqctl_report  docker_logs_rabbitmq-bundle-docker-0
docker_exec_-t_rabbitmq_image_tag_rabbitmqctl_report        docker_logs_rabbitmq_image_tag
docker_exec_-t_rabbitmq_init_bundle_rabbitmqctl_report      docker_logs_rabbitmq_init_bundle
docker_exec_-t_rabbitmq_restart_bundle_rabbitmqctl_report   docker_logs_rabbitmq_restart_bundle

sosreport-controller-0-2020-06-05-fptrhjk/sos_commands/rabbitmq/:
docker_exec_rabbitmq-bundle-docker-0_rabbitmqctl_report  docker_logs_rabbitmq-bundle-docker-0
$
~~~
Comment 13 Pavel Moravec 2020-06-09 14:46:36 UTC 
(In reply to Takashi Kajinami from comment #12)
> I've attached sosreport generated during my testing.
> 
> I confirmed that sosreport includes expected results (though unfortunately
> docker logs shows empty),
> and there is no error recorded in sos.log .

I expect the "docker logs" being empty is not an sos issue; at least I see that:

2020-06-05 02:26:24,597 INFO: [plugin:rabbitmq] added cmd output 'docker logs rabbitmq-bundle-docker-0'
2020-06-05 02:26:24,597 INFO: [plugin:rabbitmq] added cmd output 'docker exec rabbitmq-bundle-docker-0 rabbitmqctl report'
..
2020-06-05 02:26:24,608 INFO: [plugin:rabbitmq] collecting output of 'docker logs rabbitmq-bundle-docker-0'
2020-06-05 02:26:24,657 INFO: [plugin:rabbitmq] collecting output of 'docker exec rabbitmq-bundle-docker-0 rabbitmqctl report'

and the "docker exec" command has some output - so I expect running 'docker logs rabbitmq-bundle-docker-0' directly will have an empty output as well.


Thanks for the testing done!
Comment 14 Takashi Kajinami 2020-06-09 15:08:25 UTC 
(In reply to Pavel Moravec from comment #13)
> (In reply to Takashi Kajinami from comment #12)
... 
> and the "docker exec" command has some output - so I expect running 'docker
> logs rabbitmq-bundle-docker-0' directly will have an empty output as well.

Yes. I confirmed that.
Pasting the output of docker logs command for future reference.

[heat-admin@controller-0 ~]$ sudo docker ps | grep rabbitmq
9057f34f9726        192.168.24.1:8787/rh-osbs/rhosp13-openstack-rabbitmq:pcmklatest                     "dumb-init --singl..."   2 months ago        Up 2 months                                   rabbitmq-bundle-docker-0
[heat-admin@controller-0 ~]$ sudo docker logs rabbitmq-bundle-docker-0
[heat-admin@controller-0 ~]$
Comment 17 errata-xmlrpc 2020-09-29 20:55:15 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (sos bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:4034