Bug 1470005

Summary: OSP11 -> OSP12 upgrade: openstack-swift-object-expirer.service is still running under systemd after upgrading to docker architecture
Product: Red Hat OpenStack Reporter: Marius Cornea <mcornea>
Component: openstack-tripleo-heat-templatesAssignee: Emilien Macchi <emacchi>
Status: CLOSED ERRATA QA Contact: Marius Cornea <mcornea>
Severity: urgent Docs Contact:
Priority: high    
Version: 12.0 (Pike)CC: ccamacho, dbecker, mburns, morazi, rhel-osp-director-maint, sathlang, tvignaud
Target Milestone: betaKeywords: Triaged
Target Release: 12.0 (Pike)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-7.0.0-0.20170805163046.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-13 21:40:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1399762    

Description Marius Cornea 2017-07-12 09:10:57 UTC
Description of problem:
OSP11 -> OSP12 upgrade: openstack-swift-object-expirer.service is still running under systemd after upgrading to docker architecture:

During upgrade, at step overcloud-AllNodesDeploySteps-74yhsp77to5a-AllNodesPostUpgradeSteps-tl7b4sbb2xft-ControllerDeployment_Step3-o2ybgewtnni3
we can see on the overcloud node:

[root@controller-0 heat-admin]# systemctl status openstack-swift-object-expirer.service
● openstack-swift-object-expirer.service - OpenStack Object Storage (swift) - Object Expirer
   Loaded: loaded (/usr/lib/systemd/system/openstack-swift-object-expirer.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2017-07-11 18:01:56 UTC; 15h ago
 Main PID: 86168 (swift-object-ex)
   Memory: 18.8M
   CGroup: /system.slice/openstack-swift-object-expirer.service
           └─86168 /usr/bin/python2 /usr/bin/swift-object-expirer /etc/swift/object-expirer.conf

Jul 12 09:03:42 controller-0 swift[86168]: ERROR with Account server 172.17.4.10:6002/d1 re: Trying to HEAD /v1/.expiring_objects: Connection refused (txn: tx318ebe42c99f4cb89bfee-005965e5ee)
Jul 12 09:03:42 controller-0 swift[86168]: Account HEAD returning 503 for [] (txn: tx318ebe42c99f4cb89bfee-005965e5ee)
Jul 12 09:03:42 controller-0 object-expirer[86168]: Unhandled exception: #012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/obj/expirer.py", line 175, in run_once#012    self.swift.g...nal_client.py", 
Jul 12 09:06:23 controller-0 swift[86168]: ERROR with Account server 172.17.4.10:6002/d1 re: Trying to HEAD /v1/.expiring_objects: Connection refused (txn: txe269afd014e14839afc15-005965e68f)
Jul 12 09:06:23 controller-0 swift[86168]: Account HEAD returning 503 for [] (txn: txe269afd014e14839afc15-005965e68f)
Jul 12 09:06:25 controller-0 swift[86168]: ERROR with Account server 172.17.4.10:6002/d1 re: Trying to HEAD /v1/.expiring_objects: Connection refused (txn: txbd99e615031340c5a04ce-005965e691)
Jul 12 09:06:25 controller-0 swift[86168]: Account HEAD returning 503 for [] (txn: txbd99e615031340c5a04ce-005965e691)
Jul 12 09:06:29 controller-0 swift[86168]: ERROR with Account server 172.17.4.10:6002/d1 re: Trying to HEAD /v1/.expiring_objects: Connection refused (txn: tx3ef221a9ab354f0ab23bd-005965e695)
Jul 12 09:06:29 controller-0 swift[86168]: Account HEAD returning 503 for [] (txn: tx3ef221a9ab354f0ab23bd-005965e695)
Jul 12 09:06:29 controller-0 object-expirer[86168]: Unhandled exception: #012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/obj/expirer.py", line 175, in run_once#012    self.swift.g...nal_client.py", 
Hint: Some lines were ellipsized, use -l to show in fu

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-7.0.0-0.20170628002128.el7ost.noarch

How reproducible:
100%
Steps to Reproduce:
1. Deploy OSP11
2. Upgrade to OSP12 with containerized architecture

Actual results:
openstack-swift-object-expirer.service is running under systemd.

Expected results:
openstack-swift-object-expirer.service should be running only inside containers.

Additional info:

Comment 1 Carlos Camacho 2017-07-19 09:20:15 UTC
Pushed an upstream fix.

Comment 2 Carlos Camacho 2017-08-02 08:46:06 UTC
Tested now:

[heat-admin@overcloud-controller-0 ~]$ systemctl status openstack-swift-object-expirer.service
● openstack-swift-object-expirer.service - OpenStack Object Storage (swift) - Object Expirer
   Loaded: loaded (/usr/lib/systemd/system/openstack-swift-object-expirer.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

Aug 01 20:27:01 overcloud-controller-0 object-expirer[30485]: Pass completed in 0s; 0 objects expired (txn: txc0c81742b566471dbbc8d-005980e415)
Aug 01 20:28:17 overcloud-controller-0 object-expirer[30485]: Pass beginning; 0 possible containers; 0 possible objects (txn: txa10f16e5281b45aabd6dd-005980e461)
Aug 01 20:28:18 overcloud-controller-0 object-expirer[30485]: Pass completed in 0s; 0 objects expired (txn: tx6ed12ad96997487185568-005980e461)
Aug 01 20:32:23 overcloud-controller-0 object-expirer[30485]: Pass beginning; 0 possible containers; 0 possible objects (txn: txfbd9df99089d457badd2d-005980e557)
Aug 01 20:32:23 overcloud-controller-0 object-expirer[30485]: Pass completed in 0s; 0 objects expired (txn: tx1d40d73e1893471dabbd8-005980e557)
Aug 01 20:35:21 overcloud-controller-0 object-expirer[30485]: Pass beginning; 0 possible containers; 0 possible objects (txn: txfc65897df33f49a1b69b7-005980e609)
Aug 01 20:35:21 overcloud-controller-0 object-expirer[30485]: Pass completed in 0s; 0 objects expired (txn: txaeb37ab856e4483dbcf7e-005980e609)
Aug 01 20:40:05 overcloud-controller-0 systemd[1]: Stopping OpenStack Object Storage (swift) - Object Expirer...
Aug 01 20:40:05 overcloud-controller-0 object-expirer[30485]: SIGTERM received
Aug 01 20:40:05 overcloud-controller-0 systemd[1]: Stopped OpenStack Object Storage (swift) - Object Expirer.

Comment 6 Marius Cornea 2017-11-08 16:20:32 UTC
After upgrade:

[root@controller-0 heat-admin]# systemctl status openstack-swift-object-expirer.service
● openstack-swift-object-expirer.service - OpenStack Object Storage (swift) - Object Expirer
   Loaded: loaded (/usr/lib/systemd/system/openstack-swift-object-expirer.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

Nov 08 14:53:39 controller-0 object-expirer[87639]: Pass beginning; 0 possible containers; 0 possible objects (txn: txa8b456db27f94a3991694-005a031a73)
Nov 08 14:53:39 controller-0 object-expirer[87639]: Pass completed in 0s; 0 objects expired (txn: tx37e3bb8b9c3443cfb1f29-005a031a73)
Nov 08 14:55:48 controller-0 object-expirer[87639]: Pass beginning; 0 possible containers; 0 possible objects (txn: tx1bbc3bfc894d40bfa25be-005a031af4)
Nov 08 14:55:48 controller-0 object-expirer[87639]: STDERR: ERROR:root:Error talking to memcached: 172.17.1.18:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/common/memcache...ackages/eventlet
Nov 08 14:55:48 controller-0 object-expirer[87639]: STDERR: ERROR:root:Error connecting to memcached: 172.17.1.22:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/common/memca...on2.7/site-packa
Nov 08 14:55:48 controller-0 object-expirer[87639]: STDERR: ERROR:root:Error connecting to memcached: 172.17.1.23:11211#012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/common/memca...on2.7/site-packa
Nov 08 14:55:48 controller-0 object-expirer[87639]: Pass completed in 0s; 0 objects expired (txn: tx5784b3edc4514c04be260-005a031af4)
Nov 08 14:56:42 controller-0 systemd[1]: Stopping OpenStack Object Storage (swift) - Object Expirer...
Nov 08 14:56:42 controller-0 object-expirer[87639]: SIGTERM received
Nov 08 14:56:42 controller-0 systemd[1]: Stopped OpenStack Object Storage (swift) - Object Expirer.
Hint: Some lines were ellipsized, use -l to show in full.

Comment 9 errata-xmlrpc 2017-12-13 21:40:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462