Bug 1777768

Summary: [OSP14] Packets dropped during minor update
Product: Red Hat OpenStack Reporter: Jose Luis Franco <jfrancoa>
Component: openstack-tripleo-heat-templatesAssignee: Saravanan KR <skramaja>
Status: CLOSED ERRATA QA Contact: Ronnie Rasouli <rrasouli>
Severity: high Docs Contact:
Priority: high    
Version: 14.0 (Rocky)CC: cfields, lbezdick, mburns, rrasouli, sclewis, skramaja, slinaber
Target Milestone: asyncKeywords: Triaged, ZStream
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-9.3.1-0.20190513171773.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-12-20 16:49:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1770345    

Description Jose Luis Franco 2019-11-28 09:59:02 UTC
Description of problem:

The ping test running during the OSP14 minor update procedure in the CI is reporting a 4% packages loss:

TASK [tripleo-upgrade : stop l3 agent connectivity check] **********************
task path: /home/rhos-ci/jenkins/workspace/DFG-upgrades-updates-14-from-GA-composable-ipv6-scale-up/infrared/plugins/tripleo-upgrade/infrared_plugin/roles/tripleo-upgrade/tasks/common/l3_agent_connectivity_check_stop_script.yml:2
Thursday 28 November 2019  02:22:12 +0000 (1:41:16.677)       2:10:47.192 ***** 
fatal: [undercloud-0]: FAILED! => {
    "changed": true, 
    "cmd": "source /home/stack/qe-Cloud-0rc\n /home/stack/l3_agent_stop_ping.sh", 
    "delta": "0:00:00.105060", 
    "end": "2019-11-27 21:22:13.038575", 
    "rc": 1, 
    "start": "2019-11-27 21:22:12.933515"
}

STDOUT:

6078 packets transmitted, 5802 received, +168 errors, 4% packet loss, time 6083460ms
rtt min/avg/max/mdev = 0.541/1.409/1005.763/13.211 ms, pipe 4
Ping loss higher than 1% detected


MSG:

non-zero return code

	to retry, use: --limit @/home/rhos-ci/jenkins/workspace/DFG-upgrades-updates-14-from-GA-composable-ipv6-scale-up/infrared/plugins/tripleo-upgrade/infrared_plugin/main.retry

PLAY RECAP *********************************************************************
undercloud-0               : ok=71   changed=33   unreachable=0    failed=1   

Log: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-upgrades-updates-14-from-GA-composable-ipv6-scale-up/4/artifact/.sh/ir-tripleo-overcloud-update.log

The job logs were not stored, probably due to the Jenkins restart happening yesterday evening. One more logs are collected they will be added into the BZ.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Trigger CI job https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/upgrades/view/update/job/DFG-upgrades-updates-14-from-GA-composable-ipv6-scale-up/
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Lukas Bezdicka 2019-11-29 10:49:59 UTC
With patch https://review.opendev.org/#/c/696482/ applied we got 5879 packets transmitted, 5879 received, 0% packet loss, time 5884614ms

Comment 18 Ronnie Rasouli 2019-12-08 11:56:40 UTC
12862 packets transmitted, 12860 received, 0% packet loss, time 12874402ms on ping test during the upgrade process

Comment 20 errata-xmlrpc 2019-12-20 16:49:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:4339