Bug 1896877

Summary:	OSP 16.2 CI job ipv6 vlan RHELOSP 31895 Fails
Product:	Red Hat OpenStack	Reporter:	David Rosenfeld <drosenfe>
Component:	openstack-tripleo-common	Assignee:	Kamil Sambor <ksambor>
Status:	CLOSED NOTABUG	QA Contact:	David Rosenfeld <drosenfe>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	16.2 (Train)	CC:	aschultz, ksambor, mburns, ramishra, slinaber
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-02-23 11:03:56 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description David Rosenfeld 2020-11-11 18:11:34 UTC

Description of problem: Starting in OSP16.2 jenkins job: DFG-df-deployment-16.2-virthost-3cont_1comp_3ceph_3db_2net_3msg-yes_UC_SSL-yes_OC_SSL-ceph-ipv6-vlan-RHELOSP-31895 fails due to unsuccessful pings in rqci_oc_populate_after_deploy.log stage.

Suspecting that shell script: http://git.app.eng.bos.redhat.com/git/rhos-qe-core-installer.git/tree/tripleo/ci/launch_instance_vlan_IPV6_onIRsetup.sh
needs to be updated. Although same job is still successful in 16.1 without any updates.

For now BZ should be assigned to me.


Version-Release number of selected component (if applicable): RHOS-16.2-RHEL-8-20201105.n.0


How reproducible: Every time jenkins job: DFG-df-deployment-16.2-virthost-3cont_1comp_3ceph_3db_2net_3msg-yes_UC_SSL-yes_OC_SSL-ceph-ipv6-vlan-RHELOSP-31895 has been executed


Steps to Reproduce:
1. Execute jenkins job: DFG-df-deployment-16.2-virthost-3cont_1comp_3ceph_3db_2net_3msg-yes_UC_SSL-yes_OC_SSL-ceph-ipv6-vlan-RHELOSP-31895
2.
3.

Actual results: Job fails due to ping failure in rqci_oc_populate_after_deploy.log stage. The undercloud can't ping the tenant VM that was created.


Expected results: All pings are successful and job passes.


Additional info:

Comment 1 David Rosenfeld 2020-11-11 18:12:31 UTC

Logs to a failing test(See rqci_oc_populate_after_deploy.log stage):

https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/df/view/deployment/job/DFG-df-deployment-16.2-virthost-3cont_1comp_3ceph_3db_2net_3msg-yes_UC_SSL-yes_OC_SSL-ceph-ipv6-vlan-RHELOSP-31895/4/

This is seen in log:

++ expr 59 + 1
Trying to ping 2001:db8:fd00:1000:f816:3eff:fe1d:f174
+ count=60
+ echo 'Trying to ping 2001:db8:fd00:1000:f816:3eff:fe1d:f174'
+ '[' 60 -gt 60 ']'
+ ping6 -c1 -w1 2001:db8:fd00:1000:f816:3eff:fe1d:f174
PING 2001:db8:fd00:1000:f816:3eff:fe1d:f174(2001:db8:fd00:1000:f816:3eff:fe1d:f174) 56 data bytes

--- 2001:db8:fd00:1000:f816:3eff:fe1d:f174 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

Comment 3 David Rosenfeld 2020-12-07 13:35:11 UTC

I will do reruns when fix for:  https://bugzilla.redhat.com/show_bug.cgi?id=1901916 Clean up legacy Cinder... "No route to host" in ipv6 deployments
is available. That BZ is currently in Post state. The 16.2 ipv6 jobs are still failing in Phase 2.

Comment 7 David Rosenfeld 2021-03-01 14:29:14 UTC

The jobs failed for around two months and recently started working.

In 16.1:
RHOS-16.1-RHEL-8-20210205.n.0 - Failed
RHOS-16.1-RHEL-8-20210216.n.1 - Passed

In 16.2:
RHOS-16.2-RHEL-8-20210129.n.3 - Failed
RHOS-16.2-RHEL-8-20210216.n.2 - Passed

There must have been a commit between those builds that fixed the ping failures.