Bug 1597875

Created attachment 1456315 [details]
sos-controller

Description of problem:
I have completed FFWD upgrade and noticed certain service on one of the controllers have remained on the list in failed state:

(undercloud) [stack@test1-undercloud ~]$ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo systemctl list-units --state=failed 'openstack*' 'neutron*' 'httpd' 'docker' 'ceph*'" ; done
=== 172.16.0.114 ===
  UNIT                       LOAD   ACTIVE SUB    DESCRIPTION
● neutron-server.service     loaded failed failed OpenStack Neutron Server
● openstack-nova-api.service loaded failed failed OpenStack Nova API Server

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

2 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
=== 172.16.0.106 ===
0 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
=== 172.16.0.107 ===
0 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
=== 172.16.0.113 ===
0 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
=== 172.16.0.108 ===
0 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

Rebooting that controller resolved the problem and: 
neutron-server.service
openstack-nova-api.service

are no longer showing on the failed list. This might be working as designed, and if controller reboot is required after the successful upgrade, we should probably note that in a docs. Otherwise I am attaching sosreports from undercloud and controller node with failed services

Version-Release number of selected component (if applicable):
OSP10-OSP13

How reproducible:
Only on single controller

Steps to Reproduce:
1. Upgrade OSP10->OSP13
2. Verify a list of failed openstack services with:
for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo systemctl list-units --state=failed 'openstack*' 'neutron*' 'httpd' 'docker' 'ceph*'" ; done


Actual results:
'httpd' 'docker' 'ceph*'" ; done
=== 172.16.0.114 ===
  UNIT                       LOAD   ACTIVE SUB    DESCRIPTION
● neutron-server.service     loaded failed failed OpenStack Neutron Server
● openstack-nova-api.service loaded failed failed OpenStack Nova API Server

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

Expected results:
=== 172.16.0.114 ===
0 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.


Additional info: