Bug 767575

Summary: object-storage: all the processes not getting stopped and again a start of swift gives a problem
Product: [Community] GlusterFS Reporter: Saurabh <saujain>
Component: object-storageAssignee: Junaid <junaid>
Status: CLOSED NOTABUG QA Contact:
Severity: high Docs Contact:
Priority: low    
Version: pre-releaseCC: gluster-bugs, junaid, mzywusko, vagarwal, vbellur
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-01-22 06:44:16 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Saurabh 2011-12-14 12:07:20 UTC
Description of problem:
Problem occurs when some swift processes remain and  others are killed , then doing swift-init main stop does not give an issue, but swift-init main start throws an error,

Exception: Could not bind to 0.0.0.0:6012 after trying for 30 seconds


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
[root@node131 tools]# ps -eaf | grep python
root     11419     1  1 06:49 ?        00:00:00 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
root     11420     1  0 06:49 ?        00:00:00 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server/1.conf
root     11421     1  0 06:49 ?        00:00:00 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server/1.conf
root     11422     1  0 06:49 ?        00:00:00 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server/1.conf
root     11439 11420  0 06:49 ?        00:00:00 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server/1.conf
root     11441 11422  0 06:49 ?        00:00:00 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server/1.conf
root     11442 11421  0 06:49 ?        00:00:00 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server/1.conf
root     11443 11419  0 06:49 ?        00:00:00 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
root     11621 12494  0 06:50 pts/0    00:00:00 grep python
[root@node131 tools]#
[root@node131 tools]#
[root@node131 tools]# kill -9 11422
You have new mail in /var/spool/mail/root
[root@node131 tools]# kill -9 11421
[root@node131 tools]# kill -9 11420
[root@node131 tools]# kill -9 11419

Execute curl with GET command, that will be a PASS, as the other processes are running,

[root@node131 tools]# ps -eaf | grep python
root     11439     1  0 06:49 ?        00:00:00 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server/1.conf
root     11441     1  0 06:49 ?        00:00:00 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server/1.conf
root     11442     1  0 06:49 ?        00:00:00 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server/1.conf
root     11443     1  0 06:49 ?        00:00:00 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
root     12343 12494  0 06:53 pts/0    00:00:00 grep python

[root@node131 tools]# kill -9 11439
[root@node131 tools]# kill -9 11439
-bash: kill: (11439) - No such process
[root@node131 tools]# kill -9 11441
[root@node131 tools]# swift-init main stop
Signal proxy-server  pid: 11419  signal: 15
No proxy-server running
Signal container-server  pid: 11420  signal: 15
No container-server running
Signal account-server  pid: 11421  signal: 15
No account-server running
Signal object-server  pid: 11422  signal: 15
No object-server running
[root@node131 tools]# ps -eaf | grep python
root     11442     1  0 06:49 ?        00:00:00 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server/1.conf
root     11443     1  0 06:49 ?        00:00:00 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
root     12771 12494  0 06:54 pts/0    00:00:00 grep python
[root@node131 tools]# swift-init main stop 
No proxy-server running
No container-server running
No account-server running
No object-server running
[root@node131 tools]# swift-init main start
Starting proxy-server...(/etc/swift/proxy-server.conf)
Starting container-server...(/etc/swift/container-server/1.conf)
Starting account-server...(/etc/swift/account-server/1.conf)
Starting object-server...(/etc/swift/object-server/1.conf)
Traceback (most recent call last):
  File "/usr/bin/swift-proxy-server", line 5, in <module>
    pkg_resources.run_script('swift==1.4.5', 'swift-proxy-server')
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 461, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1194, in run_script
    execfile(script_filename, namespace, namespace)
  File "/usr/lib/python2.6/site-packages/swift-1.4.5-py2.6.egg/EGG-INFO/scripts/swift-proxy-server", line 22, in <module>
    run_wsgi(conf_file, 'proxy-server', default_port=8080, **options)
  File "/usr/lib/python2.6/site-packages/swift-1.4.5-py2.6.egg/swift/common/wsgi.py", line 118, in run_wsgi
    sock = get_socket(conf, default_port=kwargs.get('default_port', 8080))
  File "/usr/lib/python2.6/site-packages/swift-1.4.5-py2.6.egg/swift/common/wsgi.py", line 84, in get_socket
    bind_addr)
Exception: Could not bind to 10.1.11.131:8080 after trying for 30 seconds

Traceback (most recent call last):
  File "/usr/bin/swift-account-server", line 5, in <module>
    pkg_resources.run_script('swift==1.4.5', 'swift-account-server')
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 461, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1194, in run_script
    execfile(script_filename, namespace, namespace)
  File "/usr/lib/python2.6/site-packages/swift-1.4.5-py2.6.egg/EGG-INFO/scripts/swift-account-server", line 22, in <module>
    run_wsgi(conf_file, 'account-server', default_port=6002, **options)
  File "/usr/lib/python2.6/site-packages/swift-1.4.5-py2.6.egg/swift/common/wsgi.py", line 118, in run_wsgi
    sock = get_socket(conf, default_port=kwargs.get('default_port', 8080))
  File "/usr/lib/python2.6/site-packages/swift-1.4.5-py2.6.egg/swift/common/wsgi.py", line 84, in get_socket
    bind_addr)
Exception: Could not bind to 0.0.0.0:6012 after trying for 30 seconds





  
Actual results:


Expected results:
According this should error should be taken at the time of executing swift-init main stop command


Additional info:

Comment 1 Saurabh 2012-04-02 09:37:23 UTC
This issue is still seen on the latest swift that we are using for testing.

Comment 2 Saurabh 2012-04-02 09:39:17 UTC
Infact I had 8 worker threads for proxy-server and I killed one proxy-server , still after issuing the swift-init main stop command , proxy-servers were still seen to be running.

Comment 3 Junaid 2012-04-13 10:10:23 UTC
To kill processes, use swift-init main stop. Individual processes not be killed. Not a blocker so moving the target milestone to later.

Comment 4 Junaid 2013-01-22 06:44:16 UTC
As mentioned in the previous comment, use swift-init main stop to kill processes.