Bug 1418010 - Overcloud upgrade to RHEL 7.3 is failing
Summary: Overcloud upgrade to RHEL 7.3 is failing
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo
Version: 8.0 (Liberty)
Hardware: Unspecified
OS: Unspecified
Target Milestone: async
: ---
Assignee: Marios Andreou
QA Contact: Arik Chernetsky
Depends On:
TreeView+ depends on / blocked
Reported: 2017-01-31 15:28 UTC by Eduard Barrera
Modified: 2020-03-11 15:44 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2017-03-08 10:38:21 UTC
Target Upstream Version:

Attachments (Terms of Use)

Description Eduard Barrera 2017-01-31 15:28:20 UTC
Description of problem:

We are updating the overcloud nodes to OSP 8 RH 7.3, now satellite can see 7.3 packages instead of 7.2. openstack is already to the latest version in overcloud. 

The inital update failed with http error 504 but the stack remained update in progres... and after 17 hours the stack was interrupted by restarting heat-engine.

To update the overcloud we do ... and after a while it returns the prompt:

cd /home/stack
openstack overcloud update stack overcloud -i -vv \
  --templates \
  -e ~/templates/bb1-overcloud-storage.yaml \
  -e ~/templates/bb1-overcloud-network.yaml \
  -e ~/templates/bb1-overcloud-hostnames.yaml \
  -e ~/templates/bb1-overcloud-parameters.yaml \
  -e ~/templates/enable-tls.yaml \
  -e ~/templates/inject-trust-anchor.yaml \
  -e ~/templates/firstboot/user-data.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e ~/templates/rhel-registration/environment-rhel-registration.yaml \

curl -g -i -X GET -H 'User-Agent: python-heatclient' -H 'Content-Type: application/json' -H 'X-Auth-Url: https://172.X.X.X:13000/v2.0' -H 'Accept: application/json' -H 'X-Auth-Token: {SHA1}aea53137ffb5d42275762b13e1c4cfdb3c9bb7c6' https://172.16.X.X:13004/v1/1ec8f1ff4c114b1b93f0015a77b52865/stacks/overcloud/d901cde4-bd16-4850-bb98-01c067eda427/resources?nested_depth=5

HTTP/1.0 504 Gateway Time-out
connection: close
content-type: text/html
cache-control: no-cache

<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.

ERROR: <html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.

clean_up UpdateOvercloud: ERROR: <html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.

END return value: 1

There heat logs were truncated before launching the update again there is no relevant ERRORs but :

2017-01-31 13:55:32.422 10577 INFO heat.engine.stack [-] Stack UPDATE FAILED (overcloud): Engine went down during stack UPDATE

its also worth tho check the file show-nested.txt (91 KB) there are various stack on failed to delete

last heat logs: 	heat-logs-2017-01-31_152418_CET.tgz

full sosreport being uploaded its bug and slow to upload

Version-Release number of selected component (if applicable):


How reproducible:

Steps to Reproduce:
1. execute the overcloud update 

Actual results:
gateway timeout

Expected results:

Additional info:

Comment 4 Bob Fournier 2017-02-01 14:20:49 UTC
Eduard - can you add the contents of /var/log/messages from the node that is having problems (controller0). Thanks.

Comment 7 Bob Fournier 2017-02-01 20:51:38 UTC
Thanks Eduard.  It would also be useful to see the status of interfaces via 
'ip a' when you logged in and "found out controller0 with all the interfaces 
down". I wasn't able to correlate that to the sosreports.

Also can you run the command "neutron agent-list" on the undercloud?  It would
be good if you can run the command both after sourcing 'stackrc' and again 
after sourcing 'overcloudrc' after the deployment finishes.

Some thoughts upon looking through the logs...

1. The debug output shows these error messages:
\nERROR bb01-ctrl0 failed to join cluster in 600 seconds\n", 
 "deploy_stderr": "Error: unable to start corosync\nError: cluster is not currently running on this node

2. I'm seeing a log of neutron connectivity related issues in /var/log/messages on 
the controller, not sure if this is just due to upgrading as other neutron 
agents may be down periodically, but there are many of these type of

14:10:26 bb01-ctrl0 ceilometer-polling: 2017-02-01 14:10:26.508 17552 ERROR ceilometer.neutron_client [-] internalURL endpoint for network service not found (these eventually went away...)

Feb  1 15:16:02 bb01-ctrl0 glance-api: 2017-02-01 15:16:02.795 19026 ERROR glance.registry.client.v1.client [req-a9091770-5ac6-4408-aadc-f5a809e7b985 13fe9767454c4ca6a2f618a1e61c878a d41ceb78a14d46b79ab4140a731d75ec - - -] Registry client request GET /images/1fd81867-561f-43dd-96e3-89fd8487c67b raised NotFound

Feb  1 15:33:12 bb01-ctrl0 neutron-lbaasv2-agent: 2017-02-01 15:33:12.741 2868 ERROR neutron.common.rpc [-] Timeout in RPC method get_ready_devices. Waiting for 47 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough

Feb  1 15:33:59 bb01-ctrl0 neutron-lbaasv2-agent: 2017-02-01 15:33:59.861 2868 ERROR neutron_lbaas.agent.agent_manager [-] Unable to retrieve ready devices

3. I don't really see a connection to this issue and the initscripts bug 
https://bugzilla.redhat.com/show_bug.cgi?id=1367580, mainly because
I don't see NM bringing down any interfaces.  In the bug we have interface
down messages:
Aug  9 20:44:44 overcloud-controller-1 nm-dispatcher: Dispatching action 'down' for br-ex

but I don't see interfaces being brought in the logs.

4. The supplied journal for NetworkManager on controller doesn't appear
to have captured any problems.

5. The update stayed at "In Progress" for until it was terminated and 
all of the resources had this message:
 UPDATE paused until Hook pre-update is cleared

Comment 32 Amit Ugol 2017-03-08 10:38:21 UTC
The status of the bug is still on new, though from the comments I see that the issue was resolved, if there is a need to track something here, please raise a new bug.

Note You need to log in before you can comment on or make changes to this bug.