Bug 1150590
| Summary: | openstack-swift services fail to start after a reboot | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Community] RDO | Reporter: | wes hayutin <whayutin> | ||||
| Component: | openstack-swift | Assignee: | Pete Zaitcev <zaitcev> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | nlevinki <nlevinki> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | unspecified | CC: | apevec, derekh, whayutin, yeylon, zaitcev | ||||
| Target Milestone: | RC | ||||||
| Target Release: | Juno | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2014-10-27 18:59:30 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
openstack-swift-2.1.0-1.fc22.noarch
/etc/swift/account-server.conf:bind_ip = 172.16.32.14
/etc/swift/account-server.conf:bind_port = 6002
/etc/swift/container-server.conf:bind_ip = 172.16.32.14
/etc/swift/container-server.conf:bind_port = 6001
/etc/swift/object-server.conf:bind_ip = 172.16.32.14
/etc/swift/object-server.conf:bind_port = 6000
/etc/swift/proxy-server.conf:bind_ip = 172.16.32.14
/etc/swift/proxy-server.conf:bind_port = 8080
Starting openstack-swift-{account|container|object|proxy} manually works,
so this is a race where Swift services start before 172.16.32.14 ip address has been assigned to the interface.
Same startup issue with IPv4 not ready with libvirtd bug 1098659 Possible solution is to add http://www.freedesktop.org/software/systemd/man/systemd.special.html#network-online.target in [Unit] After= One more solution from http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ enable /proc/sys/net/ipv4/ip_nonlocal_bind http://man7.org/linux/man-pages/man7/ip.7.html (I don't see a way to set IP_FREEBIND with eventlet.listen) I don't like those nonlocal binds, they just mask issues. What if someone else is required? For instance, syslog? DNS? I should add proper dependencies per Alan's comment #3. I thought I had them though. For the time being if someone's hit with this, use bind_ip=0.0.0.0, that should work around this, hopefuly. Wes, I'm sorry, I should've been more explicit, but does setting bind_ip=0.0.0.0 cure your problem? I need this information to know for sure if I'm trying to fix the right bug here. Setting NEEDINFO. Mental note: see also bug 1119787 which deals with network-online.target. I built the fix into Rawhide for now, see openstack-swift-2.2.0-0.2.rc1.fc22. I verified that the service fails to start without it and starts with it on a local VM. Not sure if this is a 100% test, so I'm leaving Needinfo against Wes for now. |
Created attachment 945003 [details] install and config logs Description of problem: Hard to find the exact error here, it looks like swift is logging to /var/log/messages. == Swift services == openstack-swift-proxy: failed openstack-swift-account: failed openstack-swift-container: failed openstack-swift-object: failed == Cinder services == Oct 7 22:17:14 packstack swift-account-server: Traceback (most recent call last): Oct 7 22:17:14 packstack swift-account-server: File "/usr/bin/swift-account-server", line 24, in <module> Oct 7 22:17:14 packstack swift-account-server: 'account-server', default_port=6002, **options)) Oct 7 22:17:14 packstack swift-account-server: File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 424, in run_wsgi Oct 7 22:17:14 packstack swift-account-server: sock = get_socket(conf, default_port=kwargs.get('default_port', 8080)) Oct 7 22:17:14 packstack swift-account-server: File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 163, in get_socket Oct 7 22:17:14 packstack swift-account-server: family=address_family) Oct 7 22:17:14 packstack swift-account-server: File "/usr/lib/python2.7/site-packages/eventlet/convenience.py", line 39, in listen Oct 7 22:17:14 packstack swift-account-server: sock.bind(addr) Oct 7 22:17:14 packstack swift-account-server: File "/usr/lib64/python2.7/socket.py", line 224, in meth Oct 7 22:17:14 packstack swift-account-server: return getattr(self._sock,name)(*args) Oct 7 22:17:14 packstack swift-account-server: socket.error: [Errno 99] Cannot assign requested address Oct 7 22:17:15 packstack systemd: openstack-swift-account.service: main process exited, code=exited, status=1/FAILURE Oct 7 22:17:15 packstack systemd: Unit openstack-swift-account.service entered failed state. Recreate: 1. install packstack juno on f20. 2. check services, everything should work. 3. restart the server 4. check services, notice that swift is not running workaround: After the reboot, execute openstack-service restart