Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1784907

Summary: [Update] 13z8 -> z9 : Stack update failed during ceph-upgrade step.
Product: Red Hat OpenStack Reporter: Shravan Kumar Tiwari <shtiwari>
Component: os-net-configAssignee: Saravanan KR <skramaja>
Status: CLOSED DUPLICATE QA Contact: nlevinki <nlevinki>
Severity: high Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: bfournie, cfields, cfontain, hbrock, jslagle, lbezdick, mburns, skramaja
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-08 04:52:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Shravan Kumar Tiwari 2019-12-18 16:19:26 UTC 
Description of problem:
=======================
The issue is observed when a minor update from RHOSP13 release 8 to release 9 is performed in overcloud and during the update process of ceph storage node upgrade step it failed with following error.

2019-12-16 10:32:50.851 8034 INFO heat.engine.stack [req-2b630f3b-cadb-49f0-8f2d-b886622c9caf - admin - default default] Stack UPDATE FAILED (cloud05-ComputeOvsDpdk-tiyrjhcnawhq-0-xh3fybkbhsjx): Resource CREATE failed: Error: resources.NetworkDeployment: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1 2019-12-16 10:32:50.855 8034 DEBUG heat.engine.stack [req-2b630f3b-cadb-49f0-8f2d-b886622c9caf - admin - default default] Persisting stack cloud05-ComputeOvsDpdk-tiyrjhcnawhq-0-xh3fybkbhsjx status UPDATE FAILED _send_notification_and_add_event /usr/lib/python2.7/site-packages/heat/engine/stack.py:1033 2019-12-16 10:32:50.906 8040 DEBUG heat.engine.service [req-1cab9142-1715-4d76-85ce-5390e2b05af1 - - - - -] Service 37be1df5-4879-4f20-a5e3-ee50df1dde41 is updated service_manage_report /usr/lib/python2.7/site-packages/heat/engine/service.py:2324 2019-12-16 10:32:50.920 8024 DEBUG heat.engine.scheduler [req-8567c66e-70c3-4f86-a332-0f3d199e9c29 - - - - -] Task update from ResourceGroup "ComputeOvsDpdk" [dace0446-f765-4444-b49b-c697029a9d9e] Stack "cloud05" [a43e19ed-39ce-4265-a4d8-b3459c4418d6] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-12-16 10:32:50.920 8024 DEBUG heat.engine.scheduler [req-8567c66e-70c3-4f86-a332-0f3d199e9c29 - - - - -] Task _run_to_completion from ResourceGroup "ComputeOvsDpdk" [dace0446-f765-4444-b49b-c697029a9d9e] Stack "cloud05" [a43e19ed-39ce-4265-a4d8-b3459c4418d6] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-12-16 10:32:50.924 8024 DEBUG heat.engine.scheduler [req-8567c66e-70c3-4f86-a332-0f3d199e9c29 - - - - -] Task update from ResourceGroup "ComputeOvsDpdk" [dace0446-f765-4444-b49b-c697029a9d9e] Stack "cloud05" [a43e19ed-39ce-4265-a4d8-b3459c4418d6] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-12-16 10:32:50.957 8037 DEBUG heat.engine.scheduler [req-2b630f3b-cadb-49f0-8f2d-b886622c9caf - admin - default default] Task update from TemplateResource "0" [c77d3932-671b-4431-945b-66f249762e99] Stack "cloud05-ComputeOvsDpdk-tiyrjhcnawhq" [dace0446-f765-4444-b49b-c697029a9d9e] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-12-16 10:32:50.961 8037 INFO heat.engine.resource [req-2b630f3b-cadb-49f0-8f2d-b886622c9caf - admin - default default] UPDATE: TemplateResource "0" [c77d3932-671b-4431-945b-66f249762e99] Stack "cloud05-ComputeOvsDpdk-tiyrjhcnawhq" [dace0446-f765-4444-b49b-c697029a9d9e

Version-Release number of selected component (if applicable):

RHOSP13 Z9

How reproducible:
==================
Follow the update guide of OSP13 and when executing the ceph-upgrade step (as mentioned in below link) it resulted in stack update failed.

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/keeping_red_hat_openstack_platform_updated/assembly-updating_the_overcloud#updating_all_ceph_storage_nodes

Steps to Reproduce:
1.Follow the above mentioned guide for updating RHOSP13z8 t z9
2. during the step of cpeh upgrade. It failed after following command was executed.

openstack overcloud ceph-upgrade run --templates \ --stack <your overcloudstack name>


Actual results:
=================
Overcloud stack update failed.

Expected results:
===================
Stack update should succeed.

current workaround: from the available logs it was suspected that `os-net-config` is not able to assert the real state of the nic. So the nic-config in the template file for the dpdk computes was updated with actual nic names as identified by the host OS in DPDK computes.

for ex: file /home/stack/templates/nic-configs/computeovsdpdk.yaml
is updated with follwing changes:
name: nic1  
to
name: eno50

name: nic3
to
name: ens3f1

after this change theceph-upgrade step was run again and it resulted in successfully stack update and finally the convergence also executed successfully.

Additional info: internal Analysis concluded following:
======================================================
1. Update fails during ceph storage node upgrade steps, specifically on DPDK computes nodes during.

openstack overcloud ceph-upgrade run --templates \ --stack cloud05

2. It is observed that there's something wrong with the physical-nic-to-"nic number" mapping, so during os-net-config.


Dec 16 10:33:09 cloud05com002 os-collect-config: [2019/12/16 
10:30:55 AM] [INFO] eno49 is an embedded DPDK bound nic Dec 16 
10:33:09 cloud05com002 os-collect-config: [2019/12/16 10:30:55 AM] 
[INFO] ens3f0 is an DPDK bound nic Dec 16 10:33:09 cloud05com002 
os-collect-config: [2019/12/16 10:30:55 AM] [INFO] Active nics are 
['eno1', 'eno49', 'eno49', 'eno50', 'ens3f0', 'ens3f0', 'ens3f1'] Dec 16 
10:33:09 cloud05com002 os-collect-config: [2019/12/16 10:30:55 AM] 
[INFO] nic1 mapped to: eno1 Dec 16 10:33:09 cloud05com002 os- 
collect-config: [2019/12/16 10:30:55 AM] [INFO] nic7 mapped to: ens3f1 
Dec 16 10:33:09 cloud05com002 os-collect-config: [2019/12/16 
10:30:55 AM] [INFO] nic5 mapped to: ens3f0 Dec 16 10:33:09 
cloud05com002 os-collect-config: [2019/12/16 10:30:55 AM] [INFO] 
nic2 mapped to: eno49 Dec 16 10:33:09 cloud05com002 os-collect- 
config: [2019/12/16 10:30:55 AM] [INFO] nic4 mapped to: eno50


3. Seems like DPDK NICs are counted twice - once as a "regular" interface and once as a DPDK-NIC from the DPDK configfile.

[root@cloud05com002 os-net-config]# cat /var/lib/os-net- 
config/dpdk_mapping.yaml - driver: mlx5_core   mac_address: 
98:f2:b3:d0:62:b0   name: eno49   pci_address: '0000:04:00.0' - driver: 
mlx5_core   mac_address: 04:09:73:cd:63:24   name: ens3f0   
pci_address: 0000:82:00.0


4. Following os-collect-config outcome shows that interfaces failed to comeup.

Dec 16 10:33:09 cloud05com002
os-collect-config: [2019/12/16 10:30:59 AM] [INFO] Writing config 
/etc/sysconfig/network-scripts/ifcfg-ens3f0 Dec 16 10:33:09 
cloud05com002 os-collect-config: [2019/12/16 10:30:59 AM] [INFO] 
Writing config /etc/sysconfig/network-scripts/ifcfg-eno49 Dec 16 
10:33:09 cloud05com002 os-collect-config: [2019/12/16 10:30:59 AM] 
[INFO] running ifup on interface: eno1 Dec 16 10:33:09 cloud05com002 
os-collect-config: [2019/12/16 10:31:05 AM] [INFO] running ifup on 
interface: ens3f0 Dec 16 10:33:09 cloud05com002 os-collect-config: 
[2019/12/16 10:32:07 AM] [INFO] running ifup on interface: eno49 Dec 
16 10:33:09 cloud05com002 os-collect-config: [2019/12/16 10:33:09 
AM] [ERROR] Failure(s) occurred when applying configuration Dec 16 
10:33:09 cloud05com002 os-collect-config: [2019/12/16 10:33:09 AM] 
[ERROR] stdout: Dec 16 10:33:09 cloud05com002 os-collect-config: 
Determining IP information for ens3f0... failed. Dec 16 10:33:09 
cloud05com002 os-collect-config: , stderr: Dec 16 10:33:09 
cloud05com002 os-collect-config: [2019/12/16 10:33:09 AM] [ERROR] 
stdout: Dec 16 10:33:09 cloud05com002 os-collect-config: 
Determining IP information for eno49... failed. Dec 16 10:33:09 
cloud05com002 os-collect-config: , stderr: Dec 16 10:33:09 
cloud05com002 os-collect-config: Traceback (most recent call last): 
Dec 16 10:33:09 cloud05com002 os-collect-config: File "/usr/bin/os- 
net-config", line 10, in <module> Dec 16 10:33:09 cloud05com002 os- 
collect-config: sys.exit(main()) Dec 16 10:33:09 cloud05com002 os- 
collect-config: File "/usr/lib/python2.7/site- 
packages/os_net_config/cli.py", line 343, in main Dec 16 10:33:09 
cloud05com002 os-collect-config: activate=not opts.no_activate) Dec 
16 10:33:09 cloud05com002 os-collect-config: File 
"/usr/lib/python2.7/site-packages/os_net_config/impl_ifcfg.py", line 
1707, in apply Dec 16 10:33:09 cloud05com002 os-collect-config: raise 
os_net_config.ConfigurationError(message) Dec 16 10:33:09 
cloud05com002 os-collect-config: os_net_config.ConfigurationError: 
Failure(s) occurred when applying configuration Dec 16 10:33:09 
cloud05com002 os-collect-config: + RETVAL=1 Dec 16 10:33:09 
cloud05com002 os-collect-config: + set -e Dec 16 10:33:09 
cloud05com002 os-collect-config: + [[ 1 == 2 ]] Dec 16 10:33:09 
cloud05com002 os-collect-config: + [[ 1 != 0 ]] Dec 16 10:33:09 
cloud05com002 os-collect-config: + echo 'ERROR: configuration of 
safe defaults failed.' Dec 16 10:33:09 cloud05com002 os-collect- 
config: [2019-12-16 10:33:09,329] (heat-config) [ERROR] Error running 
/var/lib/heat-config/heat-config-script/41c37bac-43b4-4d09-8f7c- 
18f1acb6b8ae.



Note: please let me know if further details or logs needed for further analysis.