| Summary: | Several Swift services fail to start (and leave no logs) - but have journal messages | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Tzach Shefi <tshefi> | ||||||||||
| Component: | rhosp-director | Assignee: | Angus Thomas <athomas> | ||||||||||
| Status: | CLOSED WORKSFORME | QA Contact: | Arik Chernetsky <achernet> | ||||||||||
| Severity: | urgent | Docs Contact: | |||||||||||
| Priority: | unspecified | ||||||||||||
| Version: | 8.0 (Liberty) | CC: | dbecker, derekh, dtantsur, jslagle, mburns, mcornea, morazi, ohochman, rbartal, rhel-osp-director-maint, sgotliv, tshefi, yeylon, zaitcev | ||||||||||
| Target Milestone: | --- | ||||||||||||
| Target Release: | 8.0 (Liberty) | ||||||||||||
| Hardware: | x86_64 | ||||||||||||
| OS: | Linux | ||||||||||||
| Whiteboard: | |||||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||||
| Doc Text: | Story Points: | --- | |||||||||||
| Clone Of: | Environment: | ||||||||||||
| Last Closed: | 2016-04-04 14:38:05 UTC | Type: | Bug | ||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||
| Documentation: | --- | CRM: | |||||||||||
| Verified Versions: | Category: | --- | |||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
| Attachments: |
|
||||||||||||
Created attachment 1139903 [details]
Logs
Added logs which figured might help, if others are needed let me know which/where.
Created attachment 1139904 [details]
node screen shot
Mar 24 06:45:08 localhost.localdomain ironic-inspector[23036]: 2016-03-24 06:45:08.460 23036 ERROR swiftclient [-] ('Connection aborted.', error(111, 'ECONNREFUSED'))
Mar 24 06:45:08 localhost.localdomain ironic-inspector[23036]: 2016-03-24 06:45:08.460 23036 ERROR swiftclient Traceback (most recent call last):
Mar 24 06:45:08 localhost.localdomain ironic-inspector[23036]: 2016-03-24 06:45:08.460 23036 ERROR swiftclient File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1390, in _retry
Mar 24 06:45:08 localhost.localdomain ironic-inspector[23036]: 2016-03-24 06:45:08.460 23036 ERROR swiftclient service_token=self.service_token, **kwargs)
Mar 24 06:45:08 localhost.localdomain ironic-inspector[23036]: 2016-03-24 06:45:08.460 23036 ERROR swiftclient File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 796, in put_container
Mar 24 06:45:08 localhost.localdomain ironic-inspector[23036]: 2016-03-24 06:45:08.460 23036 ERROR swiftclient conn.request(method, path, '', headers)
Mar 24 06:45:08 localhost.localdomain ironic-inspector[23036]: 2016-03-24 06:45:08.460 23036 ERROR swiftclient File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 250, in request
Mar 24 06:45:08 localhost.localdomain ironic-inspector[23036]: 2016-03-24 06:45:08.460 23036 ERROR swiftclient files=files, **self.requests_args)
Mar 24 06:45:08 localhost.localdomain ironic-inspector[23036]: 2016-03-24 06:45:08.460 23036 ERROR swiftclient File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 233, in _request
Mar 24 06:45:08 localhost.localdomain ironic-inspector[23036]: 2016-03-24 06:45:08.460 23036 ERROR swiftclient return self.request_session.request(*arg, **kwarg)
Mar 24 06:45:08 localhost.localdomain ironic-inspector[23036]: 2016-03-24 06:45:08.460 23036 ERROR swiftclient File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
Mar 24 06:45:08 localhost.localdomain ironic-inspector[23036]: 2016-03-24 06:45:08.460 23036 ERROR swiftclient resp = self.send(prep, **send_kwargs)
Mar 24 06:45:08 localhost.localdomain ironic-inspector[23036]: 2016-03-24 06:45:08.460 23036 ERROR swiftclient File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
Mar 24 06:45:08 localhost.localdomain ironic-inspector[23036]: 2016-03-24 06:45:08.460 23036 ERROR swiftclient r = adapter.send(request, **kwargs)
Mar 24 06:45:08 localhost.localdomain ironic-inspector[23036]: 2016-03-24 06:45:08.460 23036 ERROR swiftclient File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
Mar 24 06:45:08 localhost.localdomain ironic-inspector[23036]: 2016-03-24 06:45:08.460 23036 ERROR swiftclient raise ConnectionError(err, request=request)
Mar 24 06:45:08 localhost.localdomain ironic-inspector[23036]: 2016-03-24 06:45:08.460 23036 ERROR swiftclient ConnectionError: ('Connection aborted.', error(111, 'ECONNREFUSED'))
So it look like inspector can't connect to Swift. Could you see if Swift is up and fetch its logs?
I can't find any Swift logs on your machine, /var/log/swift seems empty, nothing in journalctl too. But I can confirm that 'swift list' hangs on your machine, so it's not really an introspection problem. Anyway, could you try restarting the undercloud and checking if problem persists? If so, could you please find the swift logs? So I ran a clean install same issue Noticed some failed swift service status [root@localhost dev]# systemctl -a | grep swift | grep failed ● openstack-swift-account-replicator.service loaded failed failed OpenStack Object Storage (swift) - Account Replicator ● openstack-swift-container-updater.service loaded failed failed OpenStack Object Storage (swift) - Container Updater ● openstack-swift-proxy.service loaded failed failed OpenStack Object Storage (swift) - Proxy Server Digging around I think I know there aren't any swift logs, looking at swift config files they are all set to info with path /dev/log Under /dev/ srw-rw-rw-. 1 root root 0 Mar 30 09:19 log So I changed swift log level to debug and has changed paths to /var/log/swift/... Restarting all swift services, they now all look up and running. Still no logs. Started another introspection also failed but still now logs. BTW on a packstack deployment a coworker installed swift logs do show up under /var/log/swift. Same dir/file permissions root:root as what I have on my undercloud server yet I don't get any logging info. Making the title clear. I'm also not sure I'm the best person to be assigned here, as I have no clues about Swift and how we manage it. Tzach, did you use the latest puddle for your clean install? Yes I did clean install from yesterday March30th's build, I'll try again on another server to see if I can reproduce again and update. just setting the needinfo back to indicate we are waiting on the results of the reproduce attempt Can't reproduce, installed a clean OSPD8 deployment on March 31 All swift services are online, also don't see ECONNREFUSED errors. rhos-release 8-director -p 2016-03-29.3 rhos-release 8 -p 2016-03-29.2 [stack@seal13 ~]$ rpm -qa | grep swift openstack-swift-object-2.5.0-2.el7ost.noarch python-swiftclient-2.6.0-1.el7ost.noarch openstack-swift-2.5.0-2.el7ost.noarch openstack-swift-container-2.5.0-2.el7ost.noarch openstack-swift-account-2.5.0-2.el7ost.noarch openstack-swift-proxy-2.5.0-2.el7ost.noarch openstack-swift-plugin-swift3-1.9-1.el7ost.noarch See #11 ^ See #11 ^ forgot to clear needinfo flag Created attachment 1143272 [details]
Journal messages for swift-account-replicator
Created attachment 1143273 [details]
Journal messages for swift-proxy
Let's check logs first. You should find openstack-swift.conf under /etc/rsyslog.d which contains the following line: local0.*;local1.*;local2.* /var/log/swift/swift.log therefore I would expect to find Swift log here ^^^. I suspect that Swift services fails because ring file is missing. According to the log: Apr 03 15:54:59 seal01.qa.lab.tlv.redhat.com swift-proxy-server[14552]: IOError: [Errno 2] No such file or directory: '/etc/swift/account.ring.gz' I guess something is missing during installation process. This appears to be a one off issue based on comment 11. Please reopen if you manage to reproduce |
Description of problem: while running introspection of BM servers, Introspection completed with errors: 01099ba6-0d4b-46a1-ade0-04b1f1bfdfe4: Unexpected exception ConnectionError during processing: ('Connection aborted.', error(111, 'ECONNREFUSED')) 3dca0605-b45d-4300-9aa4-b53280d97ca5: Unexpected exception ConnectionError during processing: ('Connection aborted.', error(111, 'ECONNREFUSED')) Version-Release number of selected component (if applicable): rhel7.2 openstack-ironic-conductor-4.2.2-4.el7ost.noarch openstack-ironic-api-4.2.2-4.el7ost.noarch openstack-ironic-common-4.2.2-4.el7ost.noarch openstack-ironic-inspector-2.2.5-1.el7ost.noarch python-ironic-inspector-client-1.2.0-6.el7ost.noarch python-ironicclient-0.8.1-1.el7ost.noarch How reproducible: Reran introspection again on same hardware, same error. Steps to Reproduce: 1. Install undercloud 2. Import instack 3. Configure boot 4. Run bulk introspection 5. Disabling iptables rerunning introspection same issue. Actual results: Fail to finish introspection Introspection completed with errors: 01099ba6-0d4b-46a1-ade0-04b1f1bfdfe4: Unexpected exception ConnectionError during processing: ('Connection aborted.', error(111, 'ECONNREFUSED')) 3dca0605-b45d-4300-9aa4-b53280d97ca5: Unexpected exception ConnectionError during processing: ('Connection aborted.', error(111, 'ECONNREFUSED')) Expected results: Introspection should complete without errors. Additional info: This is happening on both nodes, both using same Intel X520 dual port 10G NIC, I don't think its related just adding info. One node is an Intel IBM server, other node is AMD based HP DL165G7.