Bug 1525620
| Summary: | rabbitmq debug commands aren't captured on controllers in OSP 12 containerized environment | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Andreas Karis <akaris> |
| Component: | sos | Assignee: | Pavel Moravec <pmoravec> |
| Status: | CLOSED ERRATA | QA Contact: | Miroslav Hradílek <mhradile> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 7.4 | CC: | agk, apevec, bmr, gavin, lhh, lmiksik, mburns, mhradile, michele, mschuppe, plambri, pmoravec, sbradley, srevivo |
| Target Milestone: | rc | Keywords: | OtherQA, Triaged |
| Target Release: | 7.5 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | sos-3.5-5.el7 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-04-10 18:05:49 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
First quick pass at this: http://acksyn.org/files/tripleo/sos-rabbitmq-container.patch Current PR is here https://github.com/sosreport/sos/pull/1191 Martin, could you please verify the BZ against sos-3.5-5 ? A yum repository for the build of sos-3.5-5.el7 (task 15272478) is available at: http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.5/5.el7/ You can install the rpms locally by putting this .repo file in your /etc/yum.repos.d/ directory: http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.5/5.el7/sos-3.5-5.el7.repo RPMs and build logs can be found in the following locations: http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.5/5.el7/noarch/ * containerized env: [root@controller-0 ~]# yum install http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.5/5.el7/noarch/sos-3.5-5.el7.noarch.rpm Loaded plugins: product-id, search-disabled-repos, subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. sos-3.5-5.el7.noarch.rpm | 407 kB 00:00:00 Examining /var/tmp/yum-root-pEiGnm/sos-3.5-5.el7.noarch.rpm: sos-3.5-5.el7.noarch Marking /var/tmp/yum-root-pEiGnm/sos-3.5-5.el7.noarch.rpm as an update to sos-3.4-13.el7_4.noarch Resolving Dependencies --> Running transaction check ---> Package sos.noarch 0:3.4-13.el7_4 will be updated ---> Package sos.noarch 0:3.5-5.el7 will be an update --> Finished Dependency Resolution Dependencies Resolved ... Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : sos-3.5-5.el7.noarch 1/2 Cleanup : sos-3.4-13.el7_4.noarch Updated: sos.noarch 0:3.5-5.el7 Complete! [root@controller-0 ~]# sosreport -o rabbitmq sosreport (version 3.5) This command will collect diagnostic and configuration information from this Red Hat Enterprise Linux system and installed applications. An archive containing the collected information will be generated in /var/tmp/sos.Hdq17P and may be provided to a Red Hat support representative. Any information provided to Red Hat will be treated in accordance with the published support policies at: https://access.redhat.com/support/ The generated archive may contain data considered sensitive and its content should be reviewed by the originating organization before being passed to any third party. No changes will be made to system configuration. Press ENTER to continue, or CTRL-C to quit. Please enter your first initial and last name [controller-0]: Please enter the case id that you are generating this report for []: Setting up archive ... Setting up plugins ... Running plugins. Please wait ... Running 1/1: rabbitmq... Creating compressed archive... Your sosreport has been generated and saved in: /var/tmp/sosreport-controller-0-20180213152206.tar.xz The checksum is: eec444498389dca2c68898115ea6702e Please send this file to your support representative. /var/tmp/sosreport-controller-0-20180213152206/sos_commands/rabbitmq [root@controller-0 rabbitmq]# ll total 7204 -rw-r--r--. 1 root root 118 Feb 13 15:22 docker_exec_-t_rabbitmq_bootstrap_rabbitmqctl_report -rw-r--r--. 1 root root 743422 Feb 13 15:22 docker_exec_-t_rabbitmq-bundle-docker-0_rabbitmqctl_report -rw-r--r--. 1 root root 118 Feb 13 15:22 docker_exec_-t_rabbitmq_image_tag_rabbitmqctl_report -rw-r--r--. 1 root root 118 Feb 13 15:22 docker_exec_-t_rabbitmq_init_bundle_rabbitmqctl_report -rw-r--r--. 1 root root 0 Feb 13 15:22 docker_logs_rabbitmq_bootstrap -rw-r--r--. 1 root root 6616435 Feb 13 15:22 docker_logs_rabbitmq-bundle-docker-0 -rw-r--r--. 1 root root 0 Feb 13 15:22 docker_logs_rabbitmq_image_tag -rw-r--r--. 1 root root 0 Feb 13 15:22 docker_logs_rabbitmq_init_bundle [root@controller-0 rabbitmq]# head -n 25 docker_exec_-t_rabbitmq-bundle-docker-0_rabbitmqctl_report Reporting server status on {{2018,2,13},{15,22,9}} ... Status of node 'rabbit@controller-1' ... [{pid,534883}, {running_applications, [{rabbitmq_management,"RabbitMQ Management Console","3.6.5"}, {rabbitmq_management_agent,"RabbitMQ Management Agent","3.6.5"}, {rabbit,"RabbitMQ","3.6.5"}, {os_mon,"CPO CXC 138 46","2.4"}, {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.6.5"}, {webmachine,"webmachine","1.10.3"}, {mochiweb,"MochiMedia Web Server","2.13.1"}, {amqp_client,"RabbitMQ AMQP Client","3.6.5"}, {rabbit_common,[],"3.6.5"}, {rabbitmq_clusterer,"Declarative RabbitMQ clustering",[]}, {mnesia,"MNESIA CXC 138 12","4.13.4"}, {ssl,"Erlang/OTP SSL application","7.3.3.1"}, {ranch,"Socket acceptor pool for TCP protocols.","1.2.1"}, {xmerl,"XML parser","1.3.10"}, {public_key,"Public key infrastructure","1.1.1"}, {asn1,"The Erlang ASN1 compiler version 4.0.2","4.0.2"}, {inets,"INETS CXC 138 49","6.2.4.1"}, {syntax_tools,"Syntax tools","1.7"}, {compiler,"ERTS CXC 138 10","6.0.3"}, ... * non containerized env: [root@controller-1 rabbitmq]# rpm -q sos sos-3.5-5.el7.noarch [root@controller-1 rabbitmq]# pwd /var/tmp/sosreport-controller-1-20180213162550/sos_commands/rabbitmq [root@controller-1 rabbitmq]# ll total 724 -rw-r--r--. 1 root root 737411 Feb 13 16:25 rabbitmqctl_report [root@controller-1 rabbitmq]# head -n 25 rabbitmqctl_report Reporting server status on {{2018,2,13},{16,25,52}} ... Status of node 'rabbit@controller-0' ... [{pid,55855}, {running_applications,[{rabbit,"RabbitMQ","3.6.5"}, {os_mon,"CPO CXC 138 46","2.4"}, {rabbit_common,[],"3.6.5"}, {mnesia,"MNESIA CXC 138 12","4.13.4"}, {ranch,"Socket acceptor pool for TCP protocols.", "1.2.1"}, {xmerl,"XML parser","1.3.10"}, {sasl,"SASL CXC 138 11","2.7"}, {stdlib,"ERTS CXC 138 10","2.8"}, {kernel,"ERTS CXC 138 10","4.2"}]}, {os,{unix,linux}}, {erlang_version,"Erlang/OTP 18 [erts-7.3.1.3] [source] [64-bit] [smp:8:8] [async-threads:128] [hipe] [kernel-poll:true]\n"}, {memory,[{total,97693792}, {connection_readers,300128}, {connection_writers,90000}, {connection_channels,346824}, {connection_other,961280}, {queue_procs,4824328}, {queue_slave_procs,5610088}, {plugins,0}, looks good, verified! Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:0963 |
Description of problem: rabbitmq debug commands aren't captured on controllers in for OSP 12 containerized environment [root@controller-0 sosreport-controller-0-20171212164931]# find . -name '*rabbitmq*' ./sos_commands/rabbitmq ./sos_commands/rabbitmq/rabbitmqctl_report ./sos_commands/rabbitmq/rabbitmqctl_cluster_status ./sos_commands/rabbitmq/rabbitmqctl_list_policies ./var/log/pacemaker/bundles/rabbitmq-bundle-0 ./var/log/pacemaker/bundles/rabbitmq-bundle-0/rabbitmq ./var/lib/config-data/puppet-generated/rabbitmq ./var/lib/config-data/puppet-generated/rabbitmq/etc/systemd/system/rabbitmq-server.service.d ./var/lib/config-data/puppet-generated/rabbitmq/etc/rabbitmq ./var/lib/config-data/puppet-generated/rabbitmq/etc/rabbitmq/rabbitmqadmin.conf ./var/lib/config-data/puppet-generated/rabbitmq/etc/rabbitmq/rabbitmq-env.conf ./var/lib/config-data/puppet-generated/rabbitmq/etc/rabbitmq/rabbitmq.config ./var/lib/config-data/puppet-generated/rabbitmq/etc/security/limits.d/rabbitmq-server.conf ./etc/selinux/targeted/active/modules/100/rabbitmq ./etc/selinux/targeted/active/modules/400/os-rabbitmq ./etc/logrotate.d/rabbitmq-server ./etc/rabbitmq ./etc/rabbitmq/rabbitmq.config ./etc/rabbitmq/rabbitmq-env.conf ./lib/systemd/system/rabbitmq-server.service [root@controller-0 sosreport-controller-0-20171212164931]# cat ./sos_commands/rabbitmq/rabbitmqctl_cluster_status erno" ienrirto rt elromgigneart ipnrge sienn td)o _booetr"r,o{r: b<a0d.ar2g.,0[>{ l_prim_loader,check_file_result,3,[]},{init,get_boot,1,[]},{init,get_boot,2,[]},{init,do_boot,3,[]}]}} init terminating in do_boot () [root@controller-0 sosreport-controller-0-20171212164931]# docker exec -it bfab6bdc5d14 rabbitmqctl cluster_status Cluster status of node 'rabbit@controller-0' ... [{nodes,[{disc,['rabbit@controller-0','rabbit@controller-1', 'rabbit@controller-2']}]}, {running_nodes,['rabbit@controller-2','rabbit@controller-1', 'rabbit@controller-0']}, {cluster_name,<<"rabbit">>}, {partitions,[]}, {alarms,[{'rabbit@controller-2',[]}, {'rabbit@controller-1',[]}, {'rabbit@controller-0',[]}]}]