Bug 976081

Summary: signing_dir must not be /etc/swift
Product: Red Hat OpenStack Reporter: Pete Zaitcev <zaitcev>
Component: openstack-packstackAssignee: Martin Magr <mmagr>
Status: CLOSED ERRATA QA Contact: Haim <hateya>
Severity: high Docs Contact:
Priority: high    
Version: 3.0CC: ajeain, aortega, apevec, ayoung, breeler, derekh, hateya, jruzicka, mmagr, yeylon, ykaul
Target Milestone: rc   
Target Release: 3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-packstack-2013.1.1-0.20.dev632.el6ost Doc Type: Bug Fix
Doc Text:
Previously, after using PackStack to install OpenStack, one could start Swift successfully the first time. However, attempting to restart Swift after the first start failed. This has been fixed and Swift restarts correctly.
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-06-27 17:18:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
RHOS patch
derekh: review+
Signing patch mk2 none

Description Pete Zaitcev 2013-06-19 20:42:24 UTC
Description of problem:

According to data captured for bug 967631, packstack --allinone creates
the following configuration in proxy-server.conf:

[filter:authtoken]
signing_dir = /etc/swift

This results in Swift denying access to its own processes and failing
with a looping crash.

Version-Release number of selected component (if applicable):

openstack-packstack-2013.1.1-0.18.dev631.el6ost.noarch

How reproducible:

Synchronous, apparently

Steps to Reproduce:
1. packstack --allinone
2. grep signing_dir /etc/swift/proxy-server.conf

Actual results:

Swift blows up

Expected results:

Swift works

Additional info:

One insiduous problem here is that the fateful chmod occurs when Swift
starts. Therefore, you can start it ONCE, test that it works. But if
you restart it, system becomes almost inaccessible with consoles
flooded by looping crash tracebacks.

The openstack-swift RPM ships with signing_dir /tmp/something-something.
Perhaps someone thought it was not secure enough. If that is a concern,
we must package a separate directory in /var/run and use /etc/tmpfiles.d
to establish proper permissions. Note that this is different on systemd
and Upstart systems like RHEL.

Comment 2 Pete Zaitcev 2013-06-19 21:17:02 UTC
Alan pointed out that this is a problem not in Packstack as such, but in
one of upstream Puppet modules, here:
 https://github.com/stackforge/puppet-swift/blob/ee4a9d48599bce332d0d7bdf4f8c0bbb6d9c6f2e/templates/proxy/authtoken.conf.erb

Comment 3 Alan Pevec 2013-06-19 22:54:26 UTC
Proper fix is not to set signing_dir at all so upstream default applies (I didn't see any justification in puppet-swift why siging_dir

In keystoneclinet master that was recently changed to random tempdir:
 https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/middleware/auth_token.py#L296
but in RHOS3 "stable/grizzly" we still have old default ~/keystone-signing:
 https://github.com/redhat-openstack/python-keystoneclient/blob/stable/grizzly/keystoneclient/middleware/auth_token.py#L214

$HOME for swift account is /var/lib/swift but this folder is not include in the RPM, so you get:
...
  File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 306, in __init__
    os.makedirs(self.signing_dirname)
  File "/usr/lib64/python2.6/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.6/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/var/lib/swift'

/var/lib/swift should be included in the openstack-swift RPM and/or backport https://github.com/openstack/python-keystoneclient/commit/03012e641d6c2a98fbfe3780102e28a65d11a887 to "stable/grizzly".

Comment 4 Alan Pevec 2013-06-20 08:26:26 UTC
(In reply to Alan Pevec from comment #3)
> Proper fix is not to set signing_dir at all so upstream default applies (I
> didn't see any justification in puppet-swift why siging_dir

... was set at all).

Comment 5 Martin Magr 2013-06-20 08:31:27 UTC
So what do you suggest? Remove "signing_dir = /etc/swift" from authtoken.conf.erb?

Comment 6 Alan Pevec 2013-06-20 08:34:44 UTC
(In reply to Martin Magr from comment #5)
> So what do you suggest? Remove "signing_dir = /etc/swift" from
> authtoken.conf.erb?

Yes, but then also need to include /var/lib/swift in the openstack-swift RPM or update keystoneclient to include upstream fix mentioned in comment 3.

Comment 8 Martin Magr 2013-06-20 09:06:17 UTC
Created attachment 763330 [details]
RHOS patch

Comment 9 Alan Pevec 2013-06-20 11:25:15 UTC
> /var/lib/swift should be included in the openstack-swift RPM

fixed in openstack-swift-1.8.0-6.el6ost

Comment 10 Martin Magr 2013-06-20 11:34:05 UTC
Created attachment 763432 [details]
Signing patch mk2

Removed removing last empty line

Comment 11 Derek Higgins 2013-06-20 12:17:38 UTC
Comment on attachment 763330 [details]
RHOS patch

lgtm

Comment 13 Bruce Reeler 2013-06-21 04:01:33 UTC
Needinfo: Martin, could you pls read Doc Text and let me know if that's OK. Thx.

Comment 14 Haim 2013-06-24 06:15:20 UTC
verified on openstack-packstack-2013.1.1-0.20.dev632.el6ost

[root@vm-161-67 ~]# grep signing_dir /etc/swift/proxy-server.conf 
[root@vm-161-67 ~]# 

swift is running: 

[root@vm-161-67 ~]# ps aux | grep swift
root      1144  0.0  0.0 103236   832 pts/0    R+   09:15   0:00 grep swift
swift    29649  0.0  0.4 237896 19580 ?        Ss   09:01   0:00 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift    29684  0.0  0.4 238760 18384 ?        S    09:01   0:00 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift    29776  0.0  0.3 228932 14192 ?        Ss   09:01   0:00 /usr/bin/python /usr/bin/swift-account-auditor /etc/swift/account-server.conf
swift    29804  0.0  0.4 230076 15792 ?        Ss   09:01   0:00 /usr/bin/python /usr/bin/swift-account-replicator /etc/swift/account-server.conf
swift    29835  0.0  0.3 229388 14796 ?        Ss   09:01   0:00 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
swift    29854  0.0  0.3 230008 13380 ?        S    09:01   0:00 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
swift    29857  0.0  0.3 228960 14360 ?        Ss   09:01   0:00 /usr/bin/python /usr/bin/swift-account-reaper /etc/swift/account-server.conf
swift    29958  0.0  0.3 222288 14416 ?        Ss   09:02   0:00 /usr/bin/python /usr/bin/swift-object-replicator /etc/swift/object-server.conf
swift    29988  0.0  0.3 228704 14524 ?        Ss   09:02   0:00 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
swift    30000  0.0  0.3 228836 11848 ?        S    09:02   0:00 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
swift    30011  0.0  0.3 229144 15044 ?        Ss   09:02   0:00 /usr/bin/python /usr/bin/swift-object-updater /etc/swift/object-server.conf
swift    30044  0.0  0.3 227988 13900 ?        Ss   09:02   0:00 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf
swift    30053  0.0  0.2 227988 11084 ?        S    09:02   0:00 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf
swift    30142  0.0  0.3 230092 15480 ?        Ss   09:02   0:00 /usr/bin/python /usr/bin/swift-container-updater /etc/swift/container-server.conf
swift    30173  0.0  0.3 229400 14812 ?        Ss   09:02   0:00 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
swift    30192  0.0  0.3 229740 12152 ?        S    09:02   0:00 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
swift    30207  0.0  0.3 228980 14200 ?        Ss   09:02   0:00 /usr/bin/python /usr/bin/swift-container-auditor /etc/swift/container-server.conf
swift    30238  0.0  0.3 230084 15484 ?        Ss   09:02   0:00 /usr/bin/python /usr/bin/swift-container-replicator /etc/swift/container-server.conf

Comment 15 Martin Magr 2013-06-24 08:52:15 UTC
Docs are OK, thanks Bruce.

Comment 16 Alan Pevec 2013-06-25 21:39:09 UTC
BTW looks like the origin of signing_dir = /etc/swift was a bad advice in one upstream bug comment: https://bugs.launchpad.net/keystone/+bug/1036847/comments/7

Comment 17 Martin Magr 2013-06-26 07:42:44 UTC
From Adam's comment [1] it seems that correct directory should be /var/cache/swift. So what now? Correct that in packstack and let swift rpm to create such directory?

[1] https://bugs.launchpad.net/keystone/+bug/1036847/comments/10

Comment 18 Pete Zaitcev 2013-06-26 14:17:54 UTC
I think Adam is right. I never was enthusiastic about throwing signing_dir
into automatically created directories, although it works as a stopgap
measure against the major screwup with directing it to /etc/swift.

I'm going to open yet another bug against Swift. At worst we'll close it
with wontfix.

Comment 19 Pete Zaitcev 2013-06-26 14:32:14 UTC
Filed bug 978408 so Fedora is consistent with RHOS going forward.

Comment 21 errata-xmlrpc 2013-06-27 17:18:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0968.html