Bug 1018123
| Summary: | RHS-C: rhsc-log-collector never completes successfully when "collect information from hypervisors" is Yes | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Prasanth <pprakash> |
| Component: | rhsc-log-collector | Assignee: | Aravinda VK <avishwan> |
| Status: | CLOSED ERRATA | QA Contact: | Prasanth <pprakash> |
| Severity: | high | Docs Contact: | |
| Priority: | medium | ||
| Version: | 2.1 | CC: | avishwan, dpati, dtsang, knarra, mmahoney, pprakash, rhs-bugs, sharne, ssampat |
| Target Milestone: | --- | Keywords: | ZStream |
| Target Release: | RHGS 2.1.2 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | 2.1.2.0 cb9 | Doc Type: | Bug Fix |
| Doc Text: |
Previously, rhsc-log-collector command to collect data from hypervisors/servers was failing. Now, with this update, the rhsc-log-collector command completes successfully collecting logs from hypervisors/servers.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2014-02-25 07:51:17 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Fixed in version: 2.1.2.0 cb5 --no-hypervisors option is changed to --no-servers Executing #rhsc-log-collector on cb5 build is throwing some xml output and on completion, it freezes the terminal. See below:
------------------
# rhsc-log-collector
INFO: Gathering oVirt Engine information...
INFO: Gathering PostgreSQL the oVirt Engine database and log files from localhost...
Please provide the REST API password for the admin@internal oVirt Engine user (CTRL+D to skip):
<?xml version="1.0" ?>
<keyValuePair>
<link href="/api/capabilities" rel="capabilities"/>
<link href="/api/clusters" rel="clusters"/>
<link href="/api/clusters?search={query}" rel="clusters/search"/>
<link href="/api/events" rel="events"/>
<link href="/api/events;from={event_id}?search={query}" rel="events/search"/>
<link href="/api/hosts" rel="hosts"/>
<link href="/api/hosts?search={query}" rel="hosts/search"/>
<link href="/api/networks" rel="networks"/>
<link href="/api/networks?search={query}" rel="networks/search"/>
<link href="/api/roles" rel="roles"/>
<link href="/api/tags" rel="tags"/>
<link href="/api/users" rel="users"/>
<link href="/api/users?search={query}" rel="users/search"/>
<link href="/api/groups" rel="groups"/>
<link href="/api/groups?search={query}" rel="groups/search"/>
<link href="/api/domains" rel="domains"/>
<special_objects>
<link href="/api/tags/00000000-0000-0000-0000-000000000000" rel="tags/root"/>
</special_objects>
<product_info>
<name>Red Hat Storage Console</name>
<vendor>Red Hat</vendor>
<version major="3" build="0" minor="3" revision="0"/>
<full_version>2.1.2-0.21.beta1.el6_4</full_version>
</product_info>
<summary>
<hosts>
<total>2</total>
<active>1</active>
</hosts>
<users>
<total>1</total>
<active>1</active>
</users>
</summary>
<time>2013-10-28T14:55:04.111000+05:30</time>
</keyValuePair>
<?xml version="1.0" ?>
<keyValuePair>
<cluster href="/api/clusters/99408929-82cf-4dc7-a532-9d998063fa95" id="99408929-82cf-4dc7-a532-9d998063fa95">
<name>Default</name>
<description>The default server cluster</description>
<link href="/api/clusters/99408929-82cf-4dc7-a532-9d998063fa95/networks" rel="networks"/>
<link href="/api/clusters/99408929-82cf-4dc7-a532-9d998063fa95/permissions" rel="permissions"/>
<link href="/api/clusters/99408929-82cf-4dc7-a532-9d998063fa95/glustervolumes" rel="glustervolumes"/>
<link href="/api/clusters/99408929-82cf-4dc7-a532-9d998063fa95/glusterhooks" rel="glusterhooks"/>
<data_center href="/api/datacenters/5849b030-626e-47cb-ad90-3ce782d831b3" id="5849b030-626e-47cb-ad90-3ce782d831b3"/>
<memory_policy>
<overcommit percent="100"/>
<transparent_hugepages>
<enabled>true</enabled>
</transparent_hugepages>
</memory_policy>
<scheduling_policy>
<policy>none</policy>
</scheduling_policy>
<version major="3" minor="3"/>
<error_handling>
<on_error>migrate</on_error>
</error_handling>
<virt_service>false</virt_service>
<gluster_service>true</gluster_service>
<threads_as_cores>false</threads_as_cores>
<tunnel_migration>false</tunnel_migration>
<trusted_service>false</trusted_service>
<ballooning_enabled>false</ballooning_enabled>
</cluster>
<cluster href="/api/clusters/aa8ab992-7445-47e1-b330-22271f40b593" id="aa8ab992-7445-47e1-b330-22271f40b593">
<name>Delhi</name>
<description>Delhi</description>
<comment>Delhi</comment>
<link href="/api/clusters/aa8ab992-7445-47e1-b330-22271f40b593/networks" rel="networks"/>
<link href="/api/clusters/aa8ab992-7445-47e1-b330-22271f40b593/permissions" rel="permissions"/>
<link href="/api/clusters/aa8ab992-7445-47e1-b330-22271f40b593/glustervolumes" rel="glustervolumes"/>
<link href="/api/clusters/aa8ab992-7445-47e1-b330-22271f40b593/glusterhooks" rel="glusterhooks"/>
<data_center href="/api/datacenters/5849b030-626e-47cb-ad90-3ce782d831b3" id="5849b030-626e-47cb-ad90-3ce782d831b3"/>
<memory_policy>
<overcommit percent="100"/>
<transparent_hugepages>
<enabled>true</enabled>
</transparent_hugepages>
</memory_policy>
<scheduling_policy>
<policy>none</policy>
</scheduling_policy>
<version major="3" minor="3"/>
<error_handling>
<on_error>migrate</on_error>
</error_handling>
<virt_service>false</virt_service>
<gluster_service>true</gluster_service>
<threads_as_cores>false</threads_as_cores>
<tunnel_migration>false</tunnel_migration>
<trusted_service>false</trusted_service>
<ballooning_enabled>false</ballooning_enabled>
</cluster>
</keyValuePair>
<?xml version="1.0" ?>
<keyValuePair>
<host href="/api/hosts/dd070e09-4b16-4da1-9333-6f8263f19d96" id="dd070e09-4b16-4da1-9333-6f8263f19d96">
<actions>
<link href="/api/hosts/dd070e09-4b16-4da1-9333-6f8263f19d96/iscsidiscover" rel="iscsidiscover"/>
<link href="/api/hosts/dd070e09-4b16-4da1-9333-6f8263f19d96/commitnetconfig" rel="commitnetconfig"/>
<link href="/api/hosts/dd070e09-4b16-4da1-9333-6f8263f19d96/approve" rel="approve"/>
<link href="/api/hosts/dd070e09-4b16-4da1-9333-6f8263f19d96/forceselectspm" rel="forceselectspm"/>
<link href="/api/hosts/dd070e09-4b16-4da1-9333-6f8263f19d96/iscsilogin" rel="iscsilogin"/>
<link href="/api/hosts/dd070e09-4b16-4da1-9333-6f8263f19d96/deactivate" rel="deactivate"/>
<link href="/api/hosts/dd070e09-4b16-4da1-9333-6f8263f19d96/fence" rel="fence"/>
<link href="/api/hosts/dd070e09-4b16-4da1-9333-6f8263f19d96/install" rel="install"/>
<link href="/api/hosts/dd070e09-4b16-4da1-9333-6f8263f19d96/activate" rel="activate"/>
</actions>
<name>Server1</name>
<link href="/api/hosts/dd070e09-4b16-4da1-9333-6f8263f19d96/storage" rel="storage"/>
<link href="/api/hosts/dd070e09-4b16-4da1-9333-6f8263f19d96/nics" rel="nics"/>
<link href="/api/hosts/dd070e09-4b16-4da1-9333-6f8263f19d96/tags" rel="tags"/>
<link href="/api/hosts/dd070e09-4b16-4da1-9333-6f8263f19d96/permissions" rel="permissions"/>
<link href="/api/hosts/dd070e09-4b16-4da1-9333-6f8263f19d96/statistics" rel="statistics"/>
<link href="/api/hosts/dd070e09-4b16-4da1-9333-6f8263f19d96/hooks" rel="hooks"/>
<address>vm10.lab.eng.blr.redhat.com</address>
<certificate>
<organization>fedora</organization>
<subject>O=fedora,CN=vm10.lab.eng.blr.redhat.com</subject>
</certificate>
<status>
<state>up</state>
</status>
<cluster href="/api/clusters/aa8ab992-7445-47e1-b330-22271f40b593" id="aa8ab992-7445-47e1-b330-22271f40b593"/>
<port>54321</port>
<type>rhel</type>
<storage_manager priority="5">false</storage_manager>
<version major="4" full_version="vdsm-4.13.0-19.gitc2a87f5.el6rhs" build="0" minor="13" revision="0"/>
<hardware_information>
<manufacturer>Red Hat</manufacturer>
<version>6Server-6.4.0.4.el6</version>
<serial_number>00000000-0000-0000-0000-00259078DB32</serial_number>
<product_name>RHEV Hypervisor</product_name>
<uuid>e08eef0c-d7d9-4589-aa46-58472adf22f1</uuid>
<family>Red Hat Enterprise Linux</family>
</hardware_information>
<power_management type="apc">
<enabled>false</enabled>
<options/>
</power_management>
<ksm>
<enabled>false</enabled>
</ksm>
<transparent_hugepages>
<enabled>true</enabled>
</transparent_hugepages>
<iscsi>
<initiator>iqn.1994-05.com.redhat:eac9944636ff</initiator>
</iscsi>
<ssh>
<port>22</port>
<fingerprint>8a:b0:e1:06:c7:d0:35:c8:ae:39:91:ff:e9:94:61:f0</fingerprint>
</ssh>
<cpu>
<topology cores="1" threads="1" sockets="2"/>
<name>Intel Xeon E312xx (Sandy Bridge)</name>
<speed>2000.</speed>
</cpu>
<memory>8387559424</memory>
<max_scheduling_memory>7982809088</max_scheduling_memory>
<summary>
<active>0</active>
<migrating>0</migrating>
<total>0</total>
</summary>
<os type="RHEL">
<version full_version="6Server - 6.4.0.4.el6"/>
</os>
<libvirt_version major="0" full_version="libvirt-0.10.2-18.el6_4.13" build="2" minor="10" revision="0"/>
</host>
<host href="/api/hosts/9f2fc941-94a9-41ee-8f10-0dbaa01cb72e" id="9f2fc941-94a9-41ee-8f10-0dbaa01cb72e">
<actions>
<link href="/api/hosts/9f2fc941-94a9-41ee-8f10-0dbaa01cb72e/iscsidiscover" rel="iscsidiscover"/>
<link href="/api/hosts/9f2fc941-94a9-41ee-8f10-0dbaa01cb72e/commitnetconfig" rel="commitnetconfig"/>
<link href="/api/hosts/9f2fc941-94a9-41ee-8f10-0dbaa01cb72e/approve" rel="approve"/>
<link href="/api/hosts/9f2fc941-94a9-41ee-8f10-0dbaa01cb72e/forceselectspm" rel="forceselectspm"/>
<link href="/api/hosts/9f2fc941-94a9-41ee-8f10-0dbaa01cb72e/iscsilogin" rel="iscsilogin"/>
<link href="/api/hosts/9f2fc941-94a9-41ee-8f10-0dbaa01cb72e/deactivate" rel="deactivate"/>
<link href="/api/hosts/9f2fc941-94a9-41ee-8f10-0dbaa01cb72e/fence" rel="fence"/>
<link href="/api/hosts/9f2fc941-94a9-41ee-8f10-0dbaa01cb72e/install" rel="install"/>
<link href="/api/hosts/9f2fc941-94a9-41ee-8f10-0dbaa01cb72e/activate" rel="activate"/>
</actions>
<name>Server2</name>
<comment>Server2</comment>
<link href="/api/hosts/9f2fc941-94a9-41ee-8f10-0dbaa01cb72e/storage" rel="storage"/>
<link href="/api/hosts/9f2fc941-94a9-41ee-8f10-0dbaa01cb72e/nics" rel="nics"/>
<link href="/api/hosts/9f2fc941-94a9-41ee-8f10-0dbaa01cb72e/tags" rel="tags"/>
<link href="/api/hosts/9f2fc941-94a9-41ee-8f10-0dbaa01cb72e/permissions" rel="permissions"/>
<link href="/api/hosts/9f2fc941-94a9-41ee-8f10-0dbaa01cb72e/statistics" rel="statistics"/>
<link href="/api/hosts/9f2fc941-94a9-41ee-8f10-0dbaa01cb72e/hooks" rel="hooks"/>
<address>vm11.lab.eng.blr.redhat.com</address>
<certificate>
<organization>fedora</organization>
<subject>O=fedora,CN=vm11.lab.eng.blr.redhat.com</subject>
</certificate>
<status>
<state>maintenance</state>
</status>
<cluster href="/api/clusters/aa8ab992-7445-47e1-b330-22271f40b593" id="aa8ab992-7445-47e1-b330-22271f40b593"/>
<port>54321</port>
<type>rhel</type>
<storage_manager priority="5">false</storage_manager>
<version major="4" full_version="vdsm-4.13.0-19.gitc2a87f5.el6rhs" build="0" minor="13" revision="0"/>
<hardware_information>
<manufacturer>Red Hat</manufacturer>
<version>6Server-6.4.0.4.el6</version>
<serial_number>00000000-0000-0000-0000-0025907C2CA6</serial_number>
<product_name>RHEV Hypervisor</product_name>
<uuid>45d3c81a-30de-428b-962c-7a044dab335f</uuid>
<family>Red Hat Enterprise Linux</family>
</hardware_information>
<power_management type="apc">
<enabled>false</enabled>
<options/>
</power_management>
<ksm>
<enabled>false</enabled>
</ksm>
<transparent_hugepages>
<enabled>true</enabled>
</transparent_hugepages>
<iscsi>
<initiator>iqn.1994-05.com.redhat:6a6c74afac6</initiator>
</iscsi>
<ssh>
<port>22</port>
<fingerprint>57:51:7f:68:f3:f2:55:1a:4e:d9:5d:6c:20:2d:b5:a1</fingerprint>
</ssh>
<cpu>
<topology cores="1" threads="1" sockets="1"/>
<name>Intel Xeon E312xx (Sandy Bridge)</name>
<speed>2000.</speed>
</cpu>
<memory>4152360960</memory>
<max_scheduling_memory>3747610624</max_scheduling_memory>
<summary>
<active>0</active>
<migrating>0</migrating>
<total>0</total>
</summary>
<os type="RHEL">
<version full_version="6Server - 6.4.0.4.el6"/>
</os>
<libvirt_version major="0" full_version="libvirt-0.10.2-18.el6_4.14" build="2" minor="10" revision="0"/>
</host>
</keyValuePair>
About to collect information from 2 servers. Continue? (Y/n): y
INFO: Gathering information from selected servers...
INFO: collecting information from vm11.lab.eng.blr.redhat.com
INFO: collecting information from vm10.lab.eng.blr.redhat.com
INFO: Gluster logs will be collected from vm11.lab.eng.blr.redhat.com
INFO: Gluster logs will be collected from vm10.lab.eng.blr.redhat.com
INFO: finished collecting information from vm11.lab.eng.blr.redhat.com
INFO: finished collecting information from vm10.lab.eng.blr.redhat.com
Creating compressed archive...
INFO: Log files have been collected and placed in /tmp/sosreport-LogCollector-20131028145600.tar.xz.
The MD5 for this file is 5cac936593986ecac651c2dc57487c18 and its size is 102.5M
------------------
Hence, moving back to assigned.
xml output is fixed in CB9, for console freeze I am still investigating the issue which caused the freeze. As a workaround, when console is freezed, type `reset` and hit enter, which clears/resets the console. (Ref: https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/3/html/Step_by_Step_Guide/s1-navigating-washing.html) In rhevm-log-collector, added a fix to prevent hang when collecting sos reports from hosts(Ref: https://bugzilla.redhat.com/show_bug.cgi?id=1022505 and https://access.redhat.com/site/solutions/489813), which causes tty freeze sometimes. I will open separate BZ to track the terminal freeze. For a workaround we can use `reset` command as I mentioned in Comment 4. Verified.
-------------
# rhsc-log-collector
INFO: Gathering oVirt Engine information...
INFO: Gathering PostgreSQL the oVirt Engine database and log files from localhost...
Please provide the REST API password for the admin@internal oVirt Engine user (CTRL+D to skip):
About to collect information from 2 servers. Continue? (Y/n): y
INFO: Gathering information from selected servers...
INFO: collecting information from vm11.lab.eng.blr.redhat.com
INFO: Gluster logs will be collected from vm11.lab.eng.blr.redhat.com
INFO: collecting information from vm10.lab.eng.blr.redhat.com
INFO: Gluster logs will be collected from vm10.lab.eng.blr.redhat.com
INFO: finished collecting information from vm10.lab.eng.blr.redhat.com
INFO: finished collecting information from vm11.lab.eng.blr.redhat.com
Creating compressed archive...
INFO: Log files have been collected and placed in /tmp/sosreport-LogCollector-20131221155932.tar.xz.
The MD5 for this file is e4529d870d13851cfd693304b4a04655 and its size is 71.1M
-------------
I'll open a separate BZ for the terminal freeze issue.
Please review the edited DocText and signoff. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2014-0208.html |
Description of problem: rhsc-log-collector never completes successfully when Y is selected for "About to collect information from <n> hypervisors. Continue? (Y/n):" ---------- [root@vm08 ovirt-engine]# rhsc-log-collector Please provide the REST API username for oVirt Engine (CTRL+D to skip): admin@internal Please provide the REST API password for the admin@internal oVirt Engine user (CTRL+D to skip): About to collect information from 2 hypervisors. Continue? (Y/n): Y INFO: Gathering information from selected hypervisors... INFO: collecting information from vm11.lab.eng.blr.redhat.com INFO: Gluster logs will be collected from vm11.lab.eng.blr.redhat.com INFO: collecting information from vm10.lab.eng.blr.redhat.com INFO: Gluster logs will be collected from vm10.lab.eng.blr.redhat.com ^CERROR: Failed to collect logs from: vm10.lab.eng.blr.redhat.com; Killed by signal 2. ERROR: Failed to collect logs from: vm11.lab.eng.blr.redhat.com; Killed by signal 2. INFO: finished collecting information from vm11.lab.eng.blr.redhat.com Exiting on user cancel. INFO: finished collecting information from vm10.lab.eng.blr.redhat.com You have new mail in /var/spool/mail/root ------- However, if you give n, it completes successfully. See below: ---------- [root@vm08 ovirt-engine]# rhsc-log-collector Please provide the REST API username for oVirt Engine (CTRL+D to skip): admin@internal Please provide the REST API password for the admin@internal oVirt Engine user (CTRL+D to skip): About to collect information from 2 hypervisors. Continue? (Y/n): n INFO: Aborting hypervisor collection... Please provide the password for the PostgreSQL user, postgres, to dump the engine PostgreSQL database instance (CTRL+D to skip): INFO: Gathering PostgreSQL the oVirt Engine database and log files from localhost... INFO: Gathering oVirt Engine information... INFO: Log files have been collected and placed in /tmp/logcollector/sosreport-LogCollector-qaredhat.com-20131011150711-2e68.tar.xz. The MD5 for this file is bef798d0b7ca9b77bf706c75683e2e68 and its size is 7.2M ---------- Version-Release number of selected component (if applicable): ---------------------- [root@vm08 /]# rpm -qa |grep rhsc rhsc-2.1.1-0.0.3.master.el6ev.noarch rhsc-lib-2.1.1-0.0.3.master.el6ev.noarch rhsc-webadmin-portal-2.1.1-0.0.3.master.el6ev.noarch rhsc-sdk-2.1.0.0-0.bb3a.el6rhs.noarch rhsc-log-collector-2.1-0.1.el6rhs.noarch rhsc-tools-2.1.1-0.0.3.master.el6ev.noarch rhsc-backend-2.1.1-0.0.3.master.el6ev.noarch rhsc-cli-2.1.0.0-0.bb3a.el6rhs.noarch rhsc-restapi-2.1.1-0.0.3.master.el6ev.noarch rhsc-dbscripts-2.1.1-0.0.3.master.el6ev.noarch rhsc-branding-rhs-3.3.0-1.0.master.201309200500.fc18.noarch rhsc-setup-2.1.1-0.0.3.master.el6ev.noarch [root@vm10 /]# rpm -qa |grep vdsm vdsm-python-cpopen-4.12.0-178.git42c732a.el6.x86_64 vdsm-gluster-4.12.0-178.git42c732a.el6.noarch vdsm-python-4.12.0-178.git42c732a.el6.x86_64 vdsm-xmlrpc-4.12.0-178.git42c732a.el6.noarch vdsm-cli-4.12.0-178.git42c732a.el6.noarch vdsm-4.12.0-178.git42c732a.el6.x86_64 ---------------------- How reproducible: Always Steps to Reproduce: 1. In engine, execute #rhsc-log-collector It runs for ever when logs from hypervisor is also selected and even though the log files are too small. Actual results: rhsc-log-collector never completes successfully Only following is seen in engine-log-collector.log: ------------- [root@vm08 ovirt-engine]# cat engine-log-collector.log 2013-10-11 14:38:56::DEBUG::hypervisors::129::root:: API Vendor(Red Hat) API Version(3.3.0) 2013-10-11 14:38:59::INFO::engine-log-collector::824::root:: Gathering information from selected hypervisors... 2013-10-11 14:38:59::INFO::engine-log-collector::538::root:: collecting information from vm11.lab.eng.blr.redhat.com 2013-10-11 14:38:59::INFO::engine-log-collector::518::root:: Gluster logs will be collected from vm11.lab.eng.blr.redhat.com 2013-10-11 14:38:59::INFO::engine-log-collector::538::root:: collecting information from vm10.lab.eng.blr.redhat.com 2013-10-11 14:38:59::INFO::engine-log-collector::518::root:: Gluster logs will be collected from vm10.lab.eng.blr.redhat.com ------------- Expected results: rhsc-log-collector should collect the sosreports from all the hypervisors and generate the tar ball. Additional info: Sosreports of engine and 2 nodes can be downloaded from: http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1018076/sosreport-cb3Engine.08-20131011142514-d79d.tar.xz http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1018076/sosreport-server1.10-20131011142152-2fbe.tar.xz http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1018076/sosreport-server2.11-20131011142223-e429.tar.xz