Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1442595

Summary: Unnecessary swift restart on scaleout of overcloud compute nodes
Product: Red Hat OpenStack Reporter: VIKRANT <vaggarwa>
Component: puppet-tripleoAssignee: RHOS Maint <rhos-maint>
Status: CLOSED DUPLICATE QA Contact: nlevinki <nlevinki>
Severity: high Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: agurenko, aschultz, jjoyce, jschluet, slinaber, tvignaud
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-04-18 15:14:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description VIKRANT 2017-04-16 01:03:08 UTC 
Description of problem:

Unnecessary swift restart on overcloud update 

Before scaleout:

# cat ps | grep -i swift
swift     121318  0.0  0.1 261732 23792 ?        Ss   18:10   0:03 /usr/bin/python2 /usr/bin/swift-container-replicator /etc/swift/container-server.conf
swift     121352  0.0  0.1 255420 21652 ?        Ss   18:10   0:00 /usr/bin/python2 /usr/bin/swift-account-reaper /etc/swift/account-server.conf
swift     121384  0.0  0.1 244528 20940 ?        Ss   18:10   0:01 /usr/bin/python2 /usr/bin/swift-account-replicator /etc/swift/account-server.conf
swift     121547  0.0  0.1 243188 19500 ?        Ss   18:10   0:00 /usr/bin/python2 /usr/bin/swift-container-auditor /etc/swift/container-server.conf
swift     121570  0.5  0.1 267916 23080 ?        Ss   18:10   0:40 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf
swift     121584  0.5  0.1 258612 22208 ?        Ss   18:10   0:40 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf
swift     121612  0.5  0.1 259640 21976 ?        Ss   18:10   0:40 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf
swift     121632  0.0  0.1 270548 23516 ?        S    18:10   0:05 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf
swift     121636  0.4  0.2 1739956 37060 ?       Ssl  18:10   0:29 /usr/bin/python2 /usr/bin/swift-object-replicator /etc/swift/object-server.conf
swift     121681  0.0  0.1 260784 22088 ?        S    18:10   0:01 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf
swift     121682  0.0  0.1 259348 23716 ?        Ss   18:10   0:00 /usr/bin/python2 /usr/bin/swift-object-auditor /etc/swift/object-server.conf
swift     121704  0.4  0.1 1738556 31012 ?       Sl   18:10   0:33 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf
swift     121709  0.0  0.1 243248 19584 ?        Ss   18:10   0:00 /usr/bin/python2 /usr/bin/swift-account-auditor /etc/swift/account-server.conf
swift     123121  0.1  0.3 344744 51164 ?        Ss   18:10   0:09 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift     123483  0.0  0.1 257544 21792 ?        Ss   18:10   0:00 /usr/bin/python2 /usr/bin/swift-object-updater /etc/swift/object-server.conf
swift     123844  0.0  0.1 243516 19564 ?        Ss   18:10   0:00 /usr/bin/python2 /usr/bin/swift-container-updater /etc/swift/container-server.conf


After scaleout:

# cat ps | grep -i swift
swift     866640  0.1  0.1 261464 23552 ?        Ss   20:56   0:00 /usr/bin/python2 /usr/bin/swift-container-replicator /etc/swift/container-server.conf
swift     866654  0.0  0.1 255260 20944 ?        Ss   20:56   0:00 /usr/bin/python2 /usr/bin/swift-account-reaper /etc/swift/account-server.conf
swift     866691  0.0  0.1 244252 20860 ?        Ss   20:56   0:00 /usr/bin/python2 /usr/bin/swift-account-replicator /etc/swift/account-server.conf
swift     866722  0.0  0.1 242920 18836 ?        Ss   20:56   0:00 /usr/bin/python2 /usr/bin/swift-container-auditor /etc/swift/container-server.conf
swift     866740  0.7  0.1 267920 23084 ?        Ss   20:56   0:03 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf
swift     866764  0.7  0.1 258612 22208 ?        Ss   20:56   0:03 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf
swift     866788  0.6  0.1 259640 21976 ?        Ss   20:56   0:03 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf
swift     866809  0.0  0.1 270072 22424 ?        S    20:56   0:00 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf
swift     866814  1.2  0.2 1736008 32596 ?       Ssl  20:56   0:05 /usr/bin/python2 /usr/bin/swift-object-replicator /etc/swift/object-server.conf
swift     866834  0.0  0.1 260788 21876 ?        S    20:56   0:00 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf
swift     866839  0.0  0.1 259352 23584 ?        Ss   20:56   0:00 /usr/bin/python2 /usr/bin/swift-object-auditor /etc/swift/object-server.conf
swift     866864  1.0  0.1 1736920 27404 ?       Sl   20:56   0:04 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf
swift     866866  0.0  0.1 242940 18792 ?        Ss   20:56   0:00 /usr/bin/python2 /usr/bin/swift-account-auditor /etc/swift/account-server.conf
swift     867166  0.1  0.3 342796 49324 ?        Ss   20:56   0:00 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift     867311  0.0  0.1 257540 21744 ?        Ss   20:56   0:00 /usr/bin/python2 /usr/bin/swift-object-updater /etc/swift/object-server.conf
swift     867325  0.0  0.1 243520 19524 ?        Ss   20:56   0:00 /usr/bin/python2 /usr/bin/swift-container-updater /etc/swift/container-server.conf
swift     912570  0.8  0.0      0     0 ?        Z    21:04   0:00 [swift-object-au] <defunct>


Version-Release number of selected component (if applicable):
RHEL OSP 10 

How reproducible:
Everytime. 

Steps to Reproduce:
1. Make a note of swift process id before scaleout. 
2. try to scaleout the compute nodes.
3. again make a note of swift process id after scaleout. All PIDs are changed that means services got restarted. 

Actual results:
Swift is getting restarted during scaleout. 

Expected results:
Swift shouldn't get restarted during scaleout. 

Additional info:
Comment 2 Alex Schultz 2017-04-17 15:46:19 UTC 
Looking at the sosreport from osp-c-1.localdomain, swift gets restarted because puppet thinks swift is not installed:

Apr 10 02:23:03 osp-c-1 os-collect-config: #033[mNotice: /Stage[main]/Swift/Package[swift]/ensure: created#033[0m

This was fixed by BZ#1426434 and according to the sos report it has not yet been applied.

puppet-tripleo-5.4.0-3.el7ost.noarch is what is installed. This issue was fixed in puppet-tripleo-5.4.0-4.el7ost.noarch

*** This bug has been marked as a duplicate of bug 1426434 ***
Comment 3 VIKRANT 2017-04-18 03:49:01 UTC 
Alex,

Thanks for the update.

Sorry, I didn't notice the version in Cu. environment. In my lab environment I am having "puppet-tripleo-5.5.0-4.el7ost.noarch" still swift restart is encountered. I am uploading the sosreport from controller node before and after the scaleout. 

~~~
[root@overcloud-controller-0 sosreport-after-scaleout-20170411133130]# cd ../sosreport-before-scaleout-20170411121515
[root@overcloud-controller-0 sosreport-before-scaleout-20170411121515]# cat ps | grep -i swift
swift     132923  0.0  0.2 261344 23492 ?        Ss   12:00   0:00 /usr/bin/python2 /usr/bin/swift-container-replicator /etc/swift/container-server.conf
swift     132992  0.0  0.2 255260 20936 ?        Ss   12:00   0:00 /usr/bin/python2 /usr/bin/swift-account-reaper /etc/swift/account-server.conf
swift     133022  0.0  0.2 244252 20848 ?        Ss   12:00   0:00 /usr/bin/python2 /usr/bin/swift-account-replicator /etc/swift/account-server.conf
swift     133059  0.0  0.2 242920 18824 ?        Ss   12:00   0:00 /usr/bin/python2 /usr/bin/swift-container-auditor /etc/swift/container-server.conf
swift     133098  0.5  0.2 267916 23076 ?        Rs   12:00   0:05 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf
swift     133152  0.5  0.2 258612 22208 ?        Rs   12:00   0:05 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf
swift     133340  0.5  0.2 259640 21972 ?        Rs   12:00   0:05 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf
swift     133377  0.0  0.2 259484 23924 ?        Ss   12:00   0:00 /usr/bin/python2 /usr/bin/swift-object-replicator /etc/swift/object-server.conf
swift     133390  0.0  0.2 270064 22712 ?        S    12:00   0:00 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf
swift     133444  0.0  0.2 259360 23596 ?        Ss   12:00   0:00 /usr/bin/python2 /usr/bin/swift-object-auditor /etc/swift/object-server.conf
swift     133459  0.0  0.2 260760 22000 ?        S    12:00   0:00 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf
swift     133490  0.0  0.2 242940 18792 ?        Ss   12:00   0:00 /usr/bin/python2 /usr/bin/swift-account-auditor /etc/swift/account-server.conf
swift     133500  0.0  0.2 261444 20276 ?        S    12:00   0:00 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf
swift     135241  0.1  0.5 343108 49532 ?        Ss   12:00   0:01 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift     135555  0.0  0.2 257504 21728 ?        Ss   12:00   0:00 /usr/bin/python2 /usr/bin/swift-object-updater /etc/swift/object-server.conf
swift     136127  0.0  0.2 243520 19508 ?        Ss   12:01   0:00 /usr/bin/python2 /usr/bin/swift-container-updater /etc/swift/container-server.conf
swift     210134  0.1  0.0      0     0 ?        Z    12:15   0:00 [swift-object-au] <defunct>

[root@overcloud-controller-0 ~]# cd sosreport-after-scaleout-20170411133130
[root@overcloud-controller-0 sosreport-after-scaleout-20170411133130]# cat ps | grep -i swift
swift     439577  0.0  0.2 261460 23636 ?        Ss   12:57   0:01 /usr/bin/python2 /usr/bin/swift-container-replicator /etc/swift/container-server.conf
swift     439591  0.0  0.2 255416 21624 ?        Ss   12:57   0:00 /usr/bin/python2 /usr/bin/swift-account-reaper /etc/swift/account-server.conf
swift     439633  0.0  0.2 244244 20860 ?        Ss   12:57   0:00 /usr/bin/python2 /usr/bin/swift-account-replicator /etc/swift/account-server.conf
swift     439654  0.0  0.2 243052 19392 ?        Ss   12:57   0:00 /usr/bin/python2 /usr/bin/swift-container-auditor /etc/swift/container-server.conf
swift     439674  0.5  0.2 267920 23084 ?        Ss   12:57   0:11 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf
swift     439699  0.5  0.2 258612 22208 ?        Ss   12:57   0:11 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf
swift     439739  0.5  0.2 259640 21972 ?        Ss   12:57   0:11 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf
swift     439762  0.1  0.2 270192 23060 ?        S    12:57   0:03 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf
swift     439765  0.2  0.3 950920 32904 ?        Ssl  12:57   0:04 /usr/bin/python2 /usr/bin/swift-object-replicator /etc/swift/object-server.conf
swift     439788  0.0  0.2 259360 23616 ?        Ss   12:57   0:00 /usr/bin/python2 /usr/bin/swift-object-auditor /etc/swift/object-server.conf
swift     439799  0.0  0.2 260784 21888 ?        S    12:57   0:00 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf
swift     439818  0.0  0.2 243072 19328 ?        Ss   12:57   0:00 /usr/bin/python2 /usr/bin/swift-account-auditor /etc/swift/account-server.conf
swift     439833  0.1  0.2 950488 26492 ?        Sl   12:57   0:02 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf
swift     440352  0.5  0.5 344724 51192 ?        Ss   12:58   0:12 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift     440476  0.0  0.2 257504 21752 ?        Ss   12:58   0:00 /usr/bin/python2 /usr/bin/swift-object-updater /etc/swift/object-server.conf
swift     440506  0.0  0.2 243520 19528 ?        Ss   12:58   0:00 /usr/bin/python2 /usr/bin/swift-container-updater /etc/swift/container-server.conf
~~~