Bug 1594262

Summary: FFU: ERROR (ClientException): Unexpected API Error. on nova list output after OSP10 -> OSP11
Product: Red Hat OpenStack Reporter: Chris Janiszewski <cjanisze>
Component: instack-undercloudAssignee: Jiri Stransky <jstransk>
Status: CLOSED DUPLICATE QA Contact: Arik Chernetsky <achernet>
Severity: medium Docs Contact:
Priority: medium    
Version: 11.0 (Ocata)CC: ccamacho, jstransk, mburns, mcornea
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-02 09:11:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
sosreport-parta
none
sos-report-partb none

Description Chris Janiszewski 2018-06-22 13:10:50 UTC 
Description of problem:
Upgraded undercloud OSP10 -> OSP11 and immediately after upgrade encountered:

2018-06-21 19:32:05,997 INFO: Not creating default plan "overcloud" because it already exists.
2018-06-21 19:32:06,766 INFO:
#############################################################################
Undercloud upgrade complete.

The file containing this installation's passwords is at
/home/stack/undercloud-passwords.conf.

There is also a stackrc file at /home/stack/stackrc.

These files are needed to interact with the OpenStack services, and should be
secured.

#############################################################################

[stack@chrisjffu-undercloud ~]$ nova list
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.ProgrammingError'> (HTTP 500) (Request-ID: req-d8f4a23f-c18c-4e29-a1e3-e8d5ce6134ca)

Later found out the OSP10 repos have not been entirely disabled during the upgrade.
Re-ran the undercloud upgrade with osp10 repos removed properly and it corrected the problem.
I am opening this bug for 2 reasons:
1. I want to see if there is anything we could do from to prevent users for getting into this issue
2. Make this issue searchable in case anyone else hit it during the upgrade
 


Version-Release number of selected component (if applicable):
OSP10->OSP11

How reproducible:


Steps to Reproduce:
1. Deploy OSP10 undercloud
2. Add osp11 repos but don't remove osp10
3. run openstack undercloud upgrade

Actual results:
nova list fails

Expected results:
nova list succeeds 

Additional info:
Attaching sosreport
Comment 1 Chris Janiszewski 2018-06-22 13:13:29 UTC 
Created attachment 1453729 [details]
sosreport-parta
Comment 2 Chris Janiszewski 2018-06-22 13:14:05 UTC 
Created attachment 1453730 [details]
sos-report-partb
Comment 3 Jiri Stransky 2018-06-25 15:07:21 UTC 
Chris thanks for the report, can you please share more high level info about how was the OSP 10/11 repository situation different on the 1st and 2nd upgrade attempt:

Did 1st attempt have OSP 11 repos enabled fully or somehow partially?

Did you use subscription manager or rhos-release?

In theory having OSP 10 repository enabled alongside OSP 11 shouldn't cause an issue, as the packages in OSP 11 repo would either be newer or the same, so the final set of RPMs should be the same too. That's assuming that *all* OSP 11 content was enabled.

(Triaging as medium/medium for now, as re-run of the undercloud upgrade with only OSP 11 repos worked. Let's increase the prio if we manage to reproduce the issue.)
Comment 4 Marius Cornea 2018-06-25 15:56:57 UTC 
I think this could potentially be a duplicate of BZ#1594265
Comment 5 Chris Janiszewski 2018-06-25 20:27:53 UTC 
it has been deployed with rhos release .. both repos (osp10 and osp11) have been enabled at the same time when it failed. I will have to go through that exercise multiple times in the next few days. I'll make note if I see it again.
Comment 6 Chris Janiszewski 2018-06-28 21:21:06 UTC 
Got the same error today after osp10->osp11 upgrade (undercloud). Used latest CDN repos. Resolved by workaround from BZ in comment #4

[stack@test1-undercloud ~]$ nova list
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.ProgrammingError'> (HTTP 500) (Request-ID: req-490ce0c7-4055-4a25-8bcd-adccc8a43bbd)
[stack@test1-undercloud ~]$ openstack server list
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.ProgrammingError'> (HTTP 500) (Request-ID: req-da19369a-93bf-44a7-8565-f45f38703ce9)
[stack@test1-undercloud ~]$ sudo nova-manage db sync
[stack@test1-undercloud ~]$ openstack server list
+--------------------------------------+----------------------+--------+-----------------------+------------+
| ID                                   | Name                 | Status | Networks              | Image Name |
+--------------------------------------+----------------------+--------+-----------------------+------------+
| c085c93f-8363-4224-ba48-d5a23fdc9f99 | testffu-controller-2 | ACTIVE | ctlplane=172.16.0.114 |            |
| 69b6c9c7-3545-447f-848e-968342f94bed | testffu-controller-0 | ACTIVE | ctlplane=172.16.0.106 |            |
| 8d4a79da-227f-4673-936b-7180e663cb4a | testffu-controller-1 | ACTIVE | ctlplane=172.16.0.107 |            |
| 5e662677-fca1-4a36-8f47-0509df285bfd | testffu-compute-1    | ACTIVE | ctlplane=172.16.0.113 |            |
| b854618b-0759-4ab6-9e4a-f885c93bf023 | testffu-compute-0    | ACTIVE | ctlplane=172.16.0.108 |            |
+--------------------------------------+----------------------+--------+-----------------------+------------+
Comment 7 Carlos Camacho 2018-07-02 09:11:51 UTC 

*** This bug has been marked as a duplicate of bug 1594265 ***