Description of problem: Unnecessary swift restart on overcloud update Before scaleout: # cat ps | grep -i swift swift 121318 0.0 0.1 261732 23792 ? Ss 18:10 0:03 /usr/bin/python2 /usr/bin/swift-container-replicator /etc/swift/container-server.conf swift 121352 0.0 0.1 255420 21652 ? Ss 18:10 0:00 /usr/bin/python2 /usr/bin/swift-account-reaper /etc/swift/account-server.conf swift 121384 0.0 0.1 244528 20940 ? Ss 18:10 0:01 /usr/bin/python2 /usr/bin/swift-account-replicator /etc/swift/account-server.conf swift 121547 0.0 0.1 243188 19500 ? Ss 18:10 0:00 /usr/bin/python2 /usr/bin/swift-container-auditor /etc/swift/container-server.conf swift 121570 0.5 0.1 267916 23080 ? Ss 18:10 0:40 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf swift 121584 0.5 0.1 258612 22208 ? Ss 18:10 0:40 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf swift 121612 0.5 0.1 259640 21976 ? Ss 18:10 0:40 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf swift 121632 0.0 0.1 270548 23516 ? S 18:10 0:05 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf swift 121636 0.4 0.2 1739956 37060 ? Ssl 18:10 0:29 /usr/bin/python2 /usr/bin/swift-object-replicator /etc/swift/object-server.conf swift 121681 0.0 0.1 260784 22088 ? S 18:10 0:01 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf swift 121682 0.0 0.1 259348 23716 ? Ss 18:10 0:00 /usr/bin/python2 /usr/bin/swift-object-auditor /etc/swift/object-server.conf swift 121704 0.4 0.1 1738556 31012 ? Sl 18:10 0:33 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf swift 121709 0.0 0.1 243248 19584 ? Ss 18:10 0:00 /usr/bin/python2 /usr/bin/swift-account-auditor /etc/swift/account-server.conf swift 123121 0.1 0.3 344744 51164 ? Ss 18:10 0:09 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf swift 123483 0.0 0.1 257544 21792 ? Ss 18:10 0:00 /usr/bin/python2 /usr/bin/swift-object-updater /etc/swift/object-server.conf swift 123844 0.0 0.1 243516 19564 ? Ss 18:10 0:00 /usr/bin/python2 /usr/bin/swift-container-updater /etc/swift/container-server.conf After scaleout: # cat ps | grep -i swift swift 866640 0.1 0.1 261464 23552 ? Ss 20:56 0:00 /usr/bin/python2 /usr/bin/swift-container-replicator /etc/swift/container-server.conf swift 866654 0.0 0.1 255260 20944 ? Ss 20:56 0:00 /usr/bin/python2 /usr/bin/swift-account-reaper /etc/swift/account-server.conf swift 866691 0.0 0.1 244252 20860 ? Ss 20:56 0:00 /usr/bin/python2 /usr/bin/swift-account-replicator /etc/swift/account-server.conf swift 866722 0.0 0.1 242920 18836 ? Ss 20:56 0:00 /usr/bin/python2 /usr/bin/swift-container-auditor /etc/swift/container-server.conf swift 866740 0.7 0.1 267920 23084 ? Ss 20:56 0:03 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf swift 866764 0.7 0.1 258612 22208 ? Ss 20:56 0:03 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf swift 866788 0.6 0.1 259640 21976 ? Ss 20:56 0:03 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf swift 866809 0.0 0.1 270072 22424 ? S 20:56 0:00 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf swift 866814 1.2 0.2 1736008 32596 ? Ssl 20:56 0:05 /usr/bin/python2 /usr/bin/swift-object-replicator /etc/swift/object-server.conf swift 866834 0.0 0.1 260788 21876 ? S 20:56 0:00 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf swift 866839 0.0 0.1 259352 23584 ? Ss 20:56 0:00 /usr/bin/python2 /usr/bin/swift-object-auditor /etc/swift/object-server.conf swift 866864 1.0 0.1 1736920 27404 ? Sl 20:56 0:04 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf swift 866866 0.0 0.1 242940 18792 ? Ss 20:56 0:00 /usr/bin/python2 /usr/bin/swift-account-auditor /etc/swift/account-server.conf swift 867166 0.1 0.3 342796 49324 ? Ss 20:56 0:00 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf swift 867311 0.0 0.1 257540 21744 ? Ss 20:56 0:00 /usr/bin/python2 /usr/bin/swift-object-updater /etc/swift/object-server.conf swift 867325 0.0 0.1 243520 19524 ? Ss 20:56 0:00 /usr/bin/python2 /usr/bin/swift-container-updater /etc/swift/container-server.conf swift 912570 0.8 0.0 0 0 ? Z 21:04 0:00 [swift-object-au] <defunct> Version-Release number of selected component (if applicable): RHEL OSP 10 How reproducible: Everytime. Steps to Reproduce: 1. Make a note of swift process id before scaleout. 2. try to scaleout the compute nodes. 3. again make a note of swift process id after scaleout. All PIDs are changed that means services got restarted. Actual results: Swift is getting restarted during scaleout. Expected results: Swift shouldn't get restarted during scaleout. Additional info:
Looking at the sosreport from osp-c-1.localdomain, swift gets restarted because puppet thinks swift is not installed: Apr 10 02:23:03 osp-c-1 os-collect-config: #033[mNotice: /Stage[main]/Swift/Package[swift]/ensure: created#033[0m This was fixed by BZ#1426434 and according to the sos report it has not yet been applied. puppet-tripleo-5.4.0-3.el7ost.noarch is what is installed. This issue was fixed in puppet-tripleo-5.4.0-4.el7ost.noarch *** This bug has been marked as a duplicate of bug 1426434 ***
Alex, Thanks for the update. Sorry, I didn't notice the version in Cu. environment. In my lab environment I am having "puppet-tripleo-5.5.0-4.el7ost.noarch" still swift restart is encountered. I am uploading the sosreport from controller node before and after the scaleout. ~~~ [root@overcloud-controller-0 sosreport-after-scaleout-20170411133130]# cd ../sosreport-before-scaleout-20170411121515 [root@overcloud-controller-0 sosreport-before-scaleout-20170411121515]# cat ps | grep -i swift swift 132923 0.0 0.2 261344 23492 ? Ss 12:00 0:00 /usr/bin/python2 /usr/bin/swift-container-replicator /etc/swift/container-server.conf swift 132992 0.0 0.2 255260 20936 ? Ss 12:00 0:00 /usr/bin/python2 /usr/bin/swift-account-reaper /etc/swift/account-server.conf swift 133022 0.0 0.2 244252 20848 ? Ss 12:00 0:00 /usr/bin/python2 /usr/bin/swift-account-replicator /etc/swift/account-server.conf swift 133059 0.0 0.2 242920 18824 ? Ss 12:00 0:00 /usr/bin/python2 /usr/bin/swift-container-auditor /etc/swift/container-server.conf swift 133098 0.5 0.2 267916 23076 ? Rs 12:00 0:05 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf swift 133152 0.5 0.2 258612 22208 ? Rs 12:00 0:05 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf swift 133340 0.5 0.2 259640 21972 ? Rs 12:00 0:05 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf swift 133377 0.0 0.2 259484 23924 ? Ss 12:00 0:00 /usr/bin/python2 /usr/bin/swift-object-replicator /etc/swift/object-server.conf swift 133390 0.0 0.2 270064 22712 ? S 12:00 0:00 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf swift 133444 0.0 0.2 259360 23596 ? Ss 12:00 0:00 /usr/bin/python2 /usr/bin/swift-object-auditor /etc/swift/object-server.conf swift 133459 0.0 0.2 260760 22000 ? S 12:00 0:00 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf swift 133490 0.0 0.2 242940 18792 ? Ss 12:00 0:00 /usr/bin/python2 /usr/bin/swift-account-auditor /etc/swift/account-server.conf swift 133500 0.0 0.2 261444 20276 ? S 12:00 0:00 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf swift 135241 0.1 0.5 343108 49532 ? Ss 12:00 0:01 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf swift 135555 0.0 0.2 257504 21728 ? Ss 12:00 0:00 /usr/bin/python2 /usr/bin/swift-object-updater /etc/swift/object-server.conf swift 136127 0.0 0.2 243520 19508 ? Ss 12:01 0:00 /usr/bin/python2 /usr/bin/swift-container-updater /etc/swift/container-server.conf swift 210134 0.1 0.0 0 0 ? Z 12:15 0:00 [swift-object-au] <defunct> [root@overcloud-controller-0 ~]# cd sosreport-after-scaleout-20170411133130 [root@overcloud-controller-0 sosreport-after-scaleout-20170411133130]# cat ps | grep -i swift swift 439577 0.0 0.2 261460 23636 ? Ss 12:57 0:01 /usr/bin/python2 /usr/bin/swift-container-replicator /etc/swift/container-server.conf swift 439591 0.0 0.2 255416 21624 ? Ss 12:57 0:00 /usr/bin/python2 /usr/bin/swift-account-reaper /etc/swift/account-server.conf swift 439633 0.0 0.2 244244 20860 ? Ss 12:57 0:00 /usr/bin/python2 /usr/bin/swift-account-replicator /etc/swift/account-server.conf swift 439654 0.0 0.2 243052 19392 ? Ss 12:57 0:00 /usr/bin/python2 /usr/bin/swift-container-auditor /etc/swift/container-server.conf swift 439674 0.5 0.2 267920 23084 ? Ss 12:57 0:11 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf swift 439699 0.5 0.2 258612 22208 ? Ss 12:57 0:11 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf swift 439739 0.5 0.2 259640 21972 ? Ss 12:57 0:11 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf swift 439762 0.1 0.2 270192 23060 ? S 12:57 0:03 /usr/bin/python2 /usr/bin/swift-container-server /etc/swift/container-server.conf swift 439765 0.2 0.3 950920 32904 ? Ssl 12:57 0:04 /usr/bin/python2 /usr/bin/swift-object-replicator /etc/swift/object-server.conf swift 439788 0.0 0.2 259360 23616 ? Ss 12:57 0:00 /usr/bin/python2 /usr/bin/swift-object-auditor /etc/swift/object-server.conf swift 439799 0.0 0.2 260784 21888 ? S 12:57 0:00 /usr/bin/python2 /usr/bin/swift-account-server /etc/swift/account-server.conf swift 439818 0.0 0.2 243072 19328 ? Ss 12:57 0:00 /usr/bin/python2 /usr/bin/swift-account-auditor /etc/swift/account-server.conf swift 439833 0.1 0.2 950488 26492 ? Sl 12:57 0:02 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf swift 440352 0.5 0.5 344724 51192 ? Ss 12:58 0:12 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf swift 440476 0.0 0.2 257504 21752 ? Ss 12:58 0:00 /usr/bin/python2 /usr/bin/swift-object-updater /etc/swift/object-server.conf swift 440506 0.0 0.2 243520 19528 ? Ss 12:58 0:00 /usr/bin/python2 /usr/bin/swift-container-updater /etc/swift/container-server.conf ~~~