| Summary: | oVirt-Engine-LogCollector - Can't collect logs from hosts: "Identity file /etc/pki/engine/keys/engine_id_rsa not accessible: No such file or directory." | ||
|---|---|---|---|
| Product: | [Retired] oVirt | Reporter: | Ortal <ogvura> |
| Component: | ovirt-engine-logcollector | Assignee: | Alex Lourie <alourie> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | unspecified | CC: | acathrow, iheim, kroberts, oschreib, ykaul |
| Target Milestone: | --- | ||
| Target Release: | 3.1 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2012-08-09 08:02:35 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
* Fixed default ssh keys path for logcollector configuration file To verify: Check that 'engine-log-collector collector all' works. You also need to patch the global variable in the logcollector itself. This variable is used then the conf file cannot be found. DEFAULT_SSH_KEY = "/etc/pki/ovirt-engine/keys/engine_id_rsa" Is there a gerrit.ovirt patch for this? Keith Take a look at http://gerrit.ovirt.org/1273 closing ON_QA bugs as oVirt 3.1 was released: http://www.ovirt.org/get-ovirt/ closing ON_QA bugs as oVirt 3.1 was released: http://www.ovirt.org/get-ovirt/ |
Description of problem: When attempting to collect logs from all host using the log collector, we can see that the log collector points to a non-existing folder: /etc/pki/engine/keys/... Where is should point is to: /etc/pki/ovirt-engine/keys Version-Release number of selected component (if applicable): ovirt-engine-jbossas-1.2-2.fc16.x86_64 How reproducible: Always Steps to Reproduce: 1. Run engine-log-collector collect all 2. Accept to collect all hosts logs. Actual results: [root@zvi-upstream ~]# engine-log-collector collect all -v Please provide the REST API username for oVirt Engine (CTRL+D to skip): admin@internal Please provide the REST API password for the admin@internal oVirt Engine user (CTRL+D to skip): DEBUG: _fetch_from_api: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <clusters> <cluster href="/api/clusters/77767de8-4669-11e1-8687-001a4a169746" id="77767de8-4669-11e1-8687-001a4a169746"> <name>buluooio</name> <link href="/api/clusters/77767de8-4669-11e1-8687-001a4a169746/networks" rel="networks"/> <link href="/api/clusters/77767de8-4669-11e1-8687-001a4a169746/permissions" rel="permissions"/> <cpu id="Intel Conroe Family"/> <data_center href="/api/datacenters/4c6a6bfe-45ac-11e1-8f75-001a4a169746" id="4c6a6bfe-45ac-11e1-8f75-001a4a169746"/> <memory_policy> <overcommit percent="100"/> <transparent_hugepages> <enabled>true</enabled> </transparent_hugepages> </memory_policy> <scheduling_policy/> <version major="3" minor="0"/> <error_handling> <on_error>migrate</on_error> </error_handling> </cluster> <cluster href="/api/clusters/99408929-82cf-4dc7-a532-9d998063fa95" id="99408929-82cf-4dc7-a532-9d998063fa95"> <name>Default</name> <description>The default server cluster</description> <link href="/api/clusters/99408929-82cf-4dc7-a532-9d998063fa95/networks" rel="networks"/> <link href="/api/clusters/99408929-82cf-4dc7-a532-9d998063fa95/permissions" rel="permissions"/> <data_center href="/api/datacenters/4c6a6bfe-45ac-11e1-8f75-001a4a169746" id="4c6a6bfe-45ac-11e1-8f75-001a4a169746"/> <memory_policy> <overcommit percent="100"/> <transparent_hugepages> <enabled>true</enabled> </transparent_hugepages> </memory_policy> <scheduling_policy/> <version major="3" minor="0"/> <error_handling> <on_error>migrate</on_error> </error_handling> </cluster> <cluster href="/api/clusters/4060d658-45ad-11e1-8009-001a4a169746" id="4060d658-45ad-11e1-8009-001a4a169746"> <name>ZVIC</name> <link href="/api/clusters/4060d658-45ad-11e1-8009-001a4a169746/networks" rel="networks"/> <link href="/api/clusters/4060d658-45ad-11e1-8009-001a4a169746/permissions" rel="permissions"/> <cpu id="Intel Conroe Family"/> <data_center href="/api/datacenters/5d691506-2817-4044-8963-591d8bf983a5" id="5d691506-2817-4044-8963-591d8bf983a5"/> <memory_policy> <overcommit percent="200"/> <transparent_hugepages> <enabled>true</enabled> </transparent_hugepages> </memory_policy> <scheduling_policy/> <version major="3" minor="0"/> <error_handling> <on_error>migrate_highly_available</on_error> </error_handling> </cluster> </clusters> DEBUG: _fetch_from_api: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <hosts> <host href="/api/hosts/0830f838-45af-11e1-aa88-001a4a169746" id="0830f838-45af-11e1-aa88-001a4a169746"> <name>navy-vds1.qa.lab.tlv.redhat.com</name> <actions> <link href="/api/hosts/0830f838-45af-11e1-aa88-001a4a169746/fence" rel="fence"/> <link href="/api/hosts/0830f838-45af-11e1-aa88-001a4a169746/install" rel="install"/> <link href="/api/hosts/0830f838-45af-11e1-aa88-001a4a169746/activate" rel="activate"/> <link href="/api/hosts/0830f838-45af-11e1-aa88-001a4a169746/deactivate" rel="deactivate"/> <link href="/api/hosts/0830f838-45af-11e1-aa88-001a4a169746/approve" rel="approve"/> <link href="/api/hosts/0830f838-45af-11e1-aa88-001a4a169746/iscsilogin" rel="iscsilogin"/> <link href="/api/hosts/0830f838-45af-11e1-aa88-001a4a169746/iscsidiscover" rel="iscsidiscover"/> <link href="/api/hosts/0830f838-45af-11e1-aa88-001a4a169746/commitnetconfig" rel="commitnetconfig"/> </actions> <link href="/api/hosts/0830f838-45af-11e1-aa88-001a4a169746/storage" rel="storage"/> <link href="/api/hosts/0830f838-45af-11e1-aa88-001a4a169746/nics" rel="nics"/> <link href="/api/hosts/0830f838-45af-11e1-aa88-001a4a169746/tags" rel="tags"/> <link href="/api/hosts/0830f838-45af-11e1-aa88-001a4a169746/permissions" rel="permissions"/> <link href="/api/hosts/0830f838-45af-11e1-aa88-001a4a169746/statistics" rel="statistics"/> <address>navy-vds1.qa.lab.tlv.redhat.com</address> <status> <state>up</state> </status> <cluster href="/api/clusters/4060d658-45ad-11e1-8009-001a4a169746" id="4060d658-45ad-11e1-8009-001a4a169746"/> <port>54321</port> <type>rhel</type> <storage_manager priority="5">true</storage_manager> <power_management type="rsa"> <enabled>true</enabled> <address>navy-vds1-mgmt.qa.lab.tlv.redhat.com</address> <username>USERID</username> <options> <option name="port" value=""/> <option name="secure" value="true"/> </options> </power_management> <ksm> <enabled>false</enabled> </ksm> <transparent_hugepages> <enabled>true</enabled> </transparent_hugepages> <iscsi> <initiator>iqn.1994-05.com.redhat:e6e477f9dd8</initiator> </iscsi> <cpu> <topology sockets="1" cores="2"/> <name>Intel(R) Xeon(R) CPU 5130 @ 2.00GHz</name> <speed>2000</speed> </cpu> <memory>6261047296</memory> <summary> <active>0</active> <migrating>0</migrating> <total>0</total> </summary> </host> <host href="/api/hosts/57f0a924-45ad-11e1-b620-001a4a169746" id="57f0a924-45ad-11e1-b620-001a4a169746"> <name>navy-vds3.qa.lab.tlv.redhat.com</name> <actions> <link href="/api/hosts/57f0a924-45ad-11e1-b620-001a4a169746/fence" rel="fence"/> <link href="/api/hosts/57f0a924-45ad-11e1-b620-001a4a169746/install" rel="install"/> <link href="/api/hosts/57f0a924-45ad-11e1-b620-001a4a169746/activate" rel="activate"/> <link href="/api/hosts/57f0a924-45ad-11e1-b620-001a4a169746/deactivate" rel="deactivate"/> <link href="/api/hosts/57f0a924-45ad-11e1-b620-001a4a169746/approve" rel="approve"/> <link href="/api/hosts/57f0a924-45ad-11e1-b620-001a4a169746/iscsilogin" rel="iscsilogin"/> <link href="/api/hosts/57f0a924-45ad-11e1-b620-001a4a169746/iscsidiscover" rel="iscsidiscover"/> <link href="/api/hosts/57f0a924-45ad-11e1-b620-001a4a169746/commitnetconfig" rel="commitnetconfig"/> </actions> <link href="/api/hosts/57f0a924-45ad-11e1-b620-001a4a169746/storage" rel="storage"/> <link href="/api/hosts/57f0a924-45ad-11e1-b620-001a4a169746/nics" rel="nics"/> <link href="/api/hosts/57f0a924-45ad-11e1-b620-001a4a169746/tags" rel="tags"/> <link href="/api/hosts/57f0a924-45ad-11e1-b620-001a4a169746/permissions" rel="permissions"/> <link href="/api/hosts/57f0a924-45ad-11e1-b620-001a4a169746/statistics" rel="statistics"/> <address>navy-vds3.qa.lab.tlv.redhat.com</address> <status> <state>maintenance</state> </status> <cluster href="/api/clusters/4060d658-45ad-11e1-8009-001a4a169746" id="4060d658-45ad-11e1-8009-001a4a169746"/> <port>54321</port> <type>rhel</type> <storage_manager priority="5">false</storage_manager> <power_management type="rsa"> <enabled>true</enabled> <address>navy-vds3-mgmt.qa.lab.tlv.redhat.com</address> <username>USERID</username> <options> <option name="port" value=""/> <option name="secure" value="false"/> </options> </power_management> <ksm> <enabled>false</enabled> </ksm> <transparent_hugepages> <enabled>true</enabled> </transparent_hugepages> <iscsi> <initiator>iqn.1994-05.com.redhat:17bfe47ba95</initiator> </iscsi> <cpu> <topology sockets="1" cores="4"/> <name>Intel(R) Xeon(R) CPU E5310 @ 1.60GHz</name> <speed>1600</speed> </cpu> <memory>4146069504</memory> <summary> <active>0</active> <migrating>0</migrating> <total>0</total> </summary> </host> </hosts> DEBUG: _fetch_from_api: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <data_centers> <data_center href="/api/datacenters/4c6a6bfe-45ac-11e1-8f75-001a4a169746" id="4c6a6bfe-45ac-11e1-8f75-001a4a169746"> <name>Default</name> <description>The default Data Center</description> <link href="/api/datacenters/4c6a6bfe-45ac-11e1-8f75-001a4a169746/storagedomains" rel="storagedomains"/> <link href="/api/datacenters/4c6a6bfe-45ac-11e1-8f75-001a4a169746/permissions" rel="permissions"/> <storage_type>iscsi</storage_type> <version major="3" minor="0"/> <supported_versions> <version major="3" minor="0"/> </supported_versions> <status> <state>uninitialized</state> </status> </data_center> <data_center href="/api/datacenters/5d691506-2817-4044-8963-591d8bf983a5" id="5d691506-2817-4044-8963-591d8bf983a5"> <name>ZVIDC</name> <description>ZVIDC</description> <link href="/api/datacenters/5d691506-2817-4044-8963-591d8bf983a5/storagedomains" rel="storagedomains"/> <link href="/api/datacenters/5d691506-2817-4044-8963-591d8bf983a5/permissions" rel="permissions"/> <storage_type>iscsi</storage_type> <storage_format>v2</storage_format> <version major="3" minor="0"/> <supported_versions> <version major="3" minor="0"/> </supported_versions> <status> <state>up</state> </status> </data_center> </data_centers> About to collect information from 2 hypervisors. Continue? (Y/n): y INFO: Gathering information from selected hypervisors... INFO: collecting information from navy-vds3.qa.lab.tlv.redhat.com INFO: collecting information from navy-vds1.qa.lab.tlv.redhat.com root.lab.tlv.redhat.com's password: root.lab.tlv.redhat.com's password: DEBUG: returncode(0) DEBUG: STDOUT( sosreport (version 2.1) This utility will collect some detailed information about the hardware and setup of your Fedora system. The information is collected and an archive is packaged under /tmp, which you can send to a support representative. Fedora will use this information for diagnostic purposes ONLY and it will be considered confidential information. This process may take a while to complete. No changes will be made to your system. Running plugins. Please wait ... Completed [11/11] ... Creating compressed archive... Your sosreport has been generated and saved in: /tmp/sosreport-navy-vds1-20120125103736-187b.tar.xz The md5sum is: ffc402701c553714c54c3b413136187b Please send this file to your support representative. ) DEBUG: STDERR(Warning: Identity file /etc/pki/engine/keys/engine_id_rsa not accessible: No such file or directory. ) DEBUG: filename(sosreport-navy-vds1-20120125103736-187b.tar.xz) DEBUG: path(/tmp/sosreport-navy-vds1-20120125103736-187b.tar.xz) DEBUG: checksum(ffc402701c553714c54c3b413136187b) root.lab.tlv.redhat.com's password: DEBUG: returncode(0) DEBUG: STDOUT( sosreport (version 2.1) This utility will collect some detailed information about the hardware and setup of your Fedora system. The information is collected and an archive is packaged under /tmp, which you can send to a support representative. Fedora will use this information for diagnostic purposes ONLY and it will be considered confidential information. This process may take a while to complete. No changes will be made to your system. Running plugins. Please wait ... Completed [11/11] ... Creating compressed archive... Your sosreport has been generated and saved in: /tmp/sosreport-navy-vds3-20120125103805-7647.tar.xz The md5sum is: d3fe5fcf5e3964c9a810478b9a017647 Please send this file to your support representative. ) DEBUG: STDERR(Warning: Identity file /etc/pki/engine/keys/engine_id_rsa not accessible: No such file or directory. ) DEBUG: filename(sosreport-navy-vds3-20120125103805-7647.tar.xz) DEBUG: path(/tmp/sosreport-navy-vds3-20120125103805-7647.tar.xz) DEBUG: checksum(d3fe5fcf5e3964c9a810478b9a017647) root.lab.tlv.redhat.com's password: DEBUG: returncode(0) DEBUG: STDOUT() DEBUG: STDERR(Warning: Identity file /etc/pki/engine/keys/engine_id_rsa not accessible: No such file or directory. ) root.lab.tlv.redhat.com's password: DEBUG: returncode(0) DEBUG: STDOUT() DEBUG: STDERR(Warning: Identity file /etc/pki/engine/keys/engine_id_rsa not accessible: No such file or directory. ) root.lab.tlv.redhat.com's password: DEBUG: returncode(0) DEBUG: STDOUT() DEBUG: STDERR(Warning: Identity file /etc/pki/engine/keys/engine_id_rsa not accessible: No such file or directory. ) root.lab.tlv.redhat.com's password: DEBUG: returncode(0) DEBUG: STDOUT() DEBUG: STDERR(Warning: Identity file /etc/pki/engine/keys/engine_id_rsa not accessible: No such file or directory. ) root.lab.tlv.redhat.com's password: DEBUG: returncode(0) DEBUG: STDOUT(Wed, 25 Jan 2012 08:38:36 +0000 ) DEBUG: STDERR(Warning: Identity file /etc/pki/engine/keys/engine_id_rsa not accessible: No such file or directory. ) DEBUG: host <navy-vds1.qa.lab.tlv.redhat.com> time: 2012-01-25T08:38:36 DEBUG: local <localhost> time: 2012-01-25T08:39:26.282401 INFO: finished collecting information from navy-vds1.qa.lab.tlv.redhat.com DEBUG: returncode(255) DEBUG: STDOUT() DEBUG: STDERR(Warning: Identity file /etc/pki/engine/keys/engine_id_rsa not accessible: No such file or directory. Connection closed by 10.35.102.23 ) ERROR: Failed to collect logs from: navy-vds3.qa.lab.tlv.redhat.com; Warning: Identity file /etc/pki/engine/keys/engine_id_rsa not accessible: No such file or directory. Connection closed by 10.35.102.23 DEBUG: Traceback (most recent call last): DEBUG: File "/usr/bin/engine-log-collector", line 462, in run DEBUG: stdout = self.caller.call('%(ssh_cmd)s "/bin/date -uR"') DEBUG: File "/usr/bin/engine-log-collector", line 84, in call DEBUG: raise Exception(stderr) DEBUG: Exception: Warning: Identity file /etc/pki/engine/keys/engine_id_rsa not accessible: No such file or directory. DEBUG: Connection closed by 10.35.102.23 DEBUG: DEBUG: Configuration for navy-vds3.qa.lab.tlv.redhat.com: DEBUG: {'archive_name': 'navy-vds3.qa.lab.tlv.redhat.com-sosreport-navy-vds3-20120125103805-7647.tar.xz', DEBUG: 'bc_reports': 'vdsm,general,networking,hardware,process,yum,filesys', DEBUG: 'checksum': 'd3fe5fcf5e3964c9a810478b9a017647', DEBUG: 'enable_jmx': False, DEBUG: 'engine': 'localhost:8443', DEBUG: 'filename': 'sosreport-navy-vds3-20120125103805-7647.tar.xz', DEBUG: 'hostname': 'navy-vds3.qa.lab.tlv.redhat.com', DEBUG: 'hosts': set([('ZVIDC', 'ZVIC', 'navy-vds1.qa.lab.tlv.redhat.com'), DEBUG: ('ZVIDC', 'ZVIC', 'navy-vds3.qa.lab.tlv.redhat.com')]), DEBUG: 'hypervisor_dir': '/tmp/logcollector/RHEVH-and-PostgreSQL-reports/navy-vds3.qa.lab.tlv.redhat.com', DEBUG: 'jboss.appxml': 'all', DEBUG: 'jboss.home': '/var/lib/jbossas', DEBUG: 'jboss.javahome': '/usr/lib/jvm/java', DEBUG: 'jboss.logsize': 15, DEBUG: 'jboss.profile': 'engine-slimmed', DEBUG: 'jboss.user': 'admin', DEBUG: 'key_file': '/etc/pki/engine/keys/engine_id_rsa', DEBUG: 'local_scratch_dir': '/tmp/logcollector/RHEVH-and-PostgreSQL-reports', DEBUG: 'local_tmp_dir': '/tmp/logcollector', DEBUG: 'log_file': '/var/log/engine/engine-log-collector.log', DEBUG: 'max_connections': 10, DEBUG: 'no_hypervisor': False, DEBUG: 'no_postgresql': False, DEBUG: 'passwd': '123456', DEBUG: 'path': '/tmp/sosreport-navy-vds3-20120125103805-7647.tar.xz', DEBUG: 'pg_dbhost': 'localhost', DEBUG: 'pg_dbname': 'engine', DEBUG: 'pg_ssh_user': 'root', DEBUG: 'pg_user': 'postgres', DEBUG: 'quiet': False, DEBUG: 'reports': 'libvirt,vdsm,general,networking,hardware,process,yum,filesys,devicemapper,selinux,kernel', DEBUG: 'scp_cmd': '/usr/bin/scp -P 22 -i /etc/pki/engine/keys/engine_id_rsa -oStrictHostKeyChecking=no root.lab.tlv.redhat.com', DEBUG: 'ssh_cmd': '/usr/bin/ssh -p 22 -i /etc/pki/engine/keys/engine_id_rsa -oStrictHostKeyChecking=no root.lab.tlv.redhat.com', DEBUG: 'ssh_port': 22, DEBUG: 'user': 'admin@internal', DEBUG: 'verbose': True} INFO: finished collecting information from navy-vds3.qa.lab.tlv.redhat.com Please provide the password for the PostgreSQL user, postgres, to dump the engine PostgreSQL database instance (CTRL+D to skip): INFO: Gathering PostgreSQL the oVirt Engine database and log files from localhost... DEBUG: returncode(0) DEBUG: STDOUT( sosreport (version 2.1) This utility will collect some detailed information about the hardware and setup of your Fedora system. The information is collected and an archive is packaged under /tmp, which you can send to a support representative. Fedora will use this information for diagnostic purposes ONLY and it will be considered confidential information. This process may take a while to complete. No changes will be made to your system. Running plugins. Please wait ... Completed [1/1] ... Creating compressed archive... Your sosreport has been generated and saved in: /tmp/logcollector/RHEVH-and-PostgreSQL-reports/sosreport-zvi-upstream-20120125104840-3c40.tar.xz The md5sum is: dd38ff7ac520b97993ffe852a23a3c40 Please send this file to your support representative. ) DEBUG: STDERR() DEBUG: filename(sosreport-zvi-upstream-20120125104840-3c40.tar.xz) DEBUG: path(/tmp/logcollector/RHEVH-and-PostgreSQL-reports/sosreport-zvi-upstream-20120125104840-3c40.tar.xz) DEBUG: checksum(dd38ff7ac520b97993ffe852a23a3c40) INFO: Gathering oVirt Engine information... DEBUG: returncode(1) DEBUG: STDOUT( sosreport (version 2.1) ) DEBUG: STDERR(unable to set option for disabled or non-existing plugin (engine) a non-existing plugin (rhevm) was specified in the command line ) ERROR: unable to set option for disabled or non-existing plugin (engine) ERROR: a non-existing plugin (rhevm) was specified in the command line Use the -h option to see usage. DEBUG: Configuration: DEBUG: command: collect DEBUG: Traceback (most recent call last): DEBUG: File "/usr/bin/engine-log-collector", line 1065, in <module> DEBUG: collector.get_engine_data() DEBUG: File "/usr/bin/engine-log-collector", line 791, in get_engine_data DEBUG: stdout = collector.sosreport() DEBUG: File "/usr/bin/engine-log-collector", line 520, in sosreport DEBUG: stdout = self.caller.call('/usr/sbin/sosreport --batch --report --tmp-dir=%(local_tmp_dir)s -o %(reports)s %(sos_options)s') DEBUG: File "/usr/bin/engine-log-collector", line 84, in call DEBUG: raise Exception(stderr) DEBUG: Exception: unable to set option for disabled or non-existing plugin (engine) DEBUG: a non-existing plugin (rhevm) was specified in the command line DEBUG: [root@zvi-upstream ~]# Expected results: The key will be found under: [root@ortal-upstream keys]# pwd /etc/pki/ovirt-engine/keys [root@ortal-upstream keys]# ll total 12 -rwxr-xr-x 1 jboss-as jboss-as 0 Jan 9 14:57 emptyfile.txt -rw------- 1 root root 916 Jan 24 16:14 engine_id_rsa -rw------- 1 root root 912 Jan 24 15:19 engine_id_rsa.old -rw-r--r-- 1 root root 220 Jan 24 16:14 engine.ssh.key.txt [root@ortal-upstream keys]#