Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 2108726

Summary: IPv6 prefix delegation system test is failing consistently
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Numan Siddique <nusiddiq>
Component: ovn22.06Assignee: Ales Musil <amusil>
Status: CLOSED ERRATA QA Contact: Jianlin Shi <jishi>
Severity: unspecified Docs Contact:
Priority: high    
Version: FDP 21.KCC: amusil, ctrautma, jiji, mmichels
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovn22.06-22.06.0-30.el8fdp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-10-11 12:25:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Patch ot reduce flakiness none

Description Numan Siddique 2022-07-19 19:33:02 UTC 
Description of problem:

Upstream test - IPv6 prefix delegation is failing consistently.

NS_EXEC_HEREDOC
system-ovn.at:5197: waiting while test "$(ovn-nbctl get logical_router_port rp-public ipv6_prefix | cut -c4-15)" = ""...
system-ovn.at:5197: wait failed after 30 seconds
./ovs-macros.at:255: hard failure
2022-07-19T16:53:14Z|00001|daemon_unix|WARN|/home/runner/work/ovn/ovn/tests/system-kmod-testsuite.dir/120/ovs-vswitchd.pid: open: No such file or directory
ovs-appctl: cannot read pidfile "/home/runner/work/ovn/ovn/tests/system-kmod-testsuite.dir/120/ovs-vswitchd.pid" (No such file or directory)
ovs-dpctl: opening datapath (No such device)
120. system-ovn.at:5197: 120. IPv6 prefix delegation -- ovn-northd -- parallelization=no -- ovn_monitor_all=no (system-ovn.at:5197): FAILED (ovs-macros.at:255)


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Ales Musil 2022-07-21 13:46:04 UTC 
The patch attached reduces the flakiness a lot, but it's still not enough. When it flakes I have noticed that dhcpd does not have a lease.
But I cannot reproduce it locally and it is impossible to debug what is going on the CI machine.
Comment 2 Ales Musil 2022-07-21 13:47:09 UTC 
Created attachment 1898480 [details]
Patch ot reduce flakiness
Comment 3 Ales Musil 2022-07-25 14:00:55 UTC 
Patches posted: https://patchwork.ozlabs.org/project/ovn/list/?series=311126
Comment 4 OVN Bot 2022-08-04 12:38:23 UTC 
ovn22.06 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2115336
Comment 8 Jianlin Shi 2022-09-21 03:54:40 UTC 
Hi Ales,

How could I reproduce the issue? I tried to run the unit test with ovn22.06-22.06.0-27, but the tests passed:

[root@dell-per740-69 ovn-22.06.0]# make check-kernel TESTSUITEFLAGS="233-240"                         
make  all-am
make[1]: Entering directory '/root/rpmbuild/BUILD/ovn-22.06.0'                                        
make[1]: Leaving directory '/root/rpmbuild/BUILD/ovn-22.06.0'                                         
set /bin/sh './tests/system-kmod-testsuite' -C tests  AUTOTEST_PATH='/root/rpmbuild/BUILD/ovn-22.06.0/ovs-6f24c2bc769afde0a390ce344de1a7d9c592e5a6/utilities:/root/rpmbuild/BUILD/ovn-22.06.0/ovs-6f24c2bc769afde0a390ce344de1a7d9c592e5a6/vswitchd:/root/rpmbuild/BUILD/ovn-22.06.0/ovs-6f24c2bc769afde0a390ce344de1a7d9c592e5a6/ovsdb:/root/rpmbuild/BUILD/ovn-22.06.0/ovs-6f24c2bc769afde0a390ce344de1a7d9c592e5a6/vtep:tests:::controller-vtep:northd:utilities:controller:ic'; \                                         
 "$@" 233-240 -j1 || (test X'' = Xyes &&  "$@" --recheck)                                             
## ----------------------- ##                                                                         
## ovn 22.06.1 test suite. ##
## ----------------------- ##

system-ovn

233: IPv6 prefix delegation -- ovn-northd -- dp-groups=yes -- parallelization=yes ok                  
234: IPv6 prefix delegation -- ovn-northd -- dp-groups=yes -- parallelization=no ok                   
235: IPv6 prefix delegation -- ovn-northd -- dp-groups=no -- parallelization=yes ok                   
236: IPv6 prefix delegation -- ovn-northd -- dp-groups=no -- parallelization=no ok                    
237: IPv6 prefix delegation -- ovn-northd-ddlog -- dp-groups=yes -- parallelization=yes skipped (system-ovn.at:5197)
238: IPv6 prefix delegation -- ovn-northd-ddlog -- dp-groups=yes -- parallelization=no skipped (system-ovn.at:5197)
239: IPv6 prefix delegation -- ovn-northd-ddlog -- dp-groups=no -- parallelization=yes skipped (system-ovn.at:5197)
240: IPv6 prefix delegation -- ovn-northd-ddlog -- dp-groups=no -- parallelization=no skipped (system-ovn.at:5197)

## ------------- ##
## Test results. ##                                                                                   
## ------------- ##                                                                                   

4 tests were successful.
4 tests were skipped.
Comment 9 Ales Musil 2022-09-21 05:00:13 UTC 
(In reply to Jianlin Shi from comment #8)
> Hi Ales,
> 
> How could I reproduce the issue? I tried to run the unit test with
> ovn22.06-22.06.0-27, but the tests passed:
> 
> [root@dell-per740-69 ovn-22.06.0]# make check-kernel
> TESTSUITEFLAGS="233-240"                         
> make  all-am
> make[1]: Entering directory '/root/rpmbuild/BUILD/ovn-22.06.0'              
> 
> make[1]: Leaving directory '/root/rpmbuild/BUILD/ovn-22.06.0'               
> 
> set /bin/sh './tests/system-kmod-testsuite' -C tests 
> AUTOTEST_PATH='/root/rpmbuild/BUILD/ovn-22.06.0/ovs-
> 6f24c2bc769afde0a390ce344de1a7d9c592e5a6/utilities:/root/rpmbuild/BUILD/ovn-
> 22.06.0/ovs-6f24c2bc769afde0a390ce344de1a7d9c592e5a6/vswitchd:/root/rpmbuild/
> BUILD/ovn-22.06.0/ovs-6f24c2bc769afde0a390ce344de1a7d9c592e5a6/ovsdb:/root/
> rpmbuild/BUILD/ovn-22.06.0/ovs-6f24c2bc769afde0a390ce344de1a7d9c592e5a6/vtep:
> tests:::controller-vtep:northd:utilities:controller:ic'; \                  
> 
>  "$@" 233-240 -j1 || (test X'' = Xyes &&  "$@" --recheck)                   
> 
> ## ----------------------- ##                                               
> 
> ## ovn 22.06.1 test suite. ##
> ## ----------------------- ##
> 
> system-ovn
> 
> 233: IPv6 prefix delegation -- ovn-northd -- dp-groups=yes --
> parallelization=yes ok                  
> 234: IPv6 prefix delegation -- ovn-northd -- dp-groups=yes --
> parallelization=no ok                   
> 235: IPv6 prefix delegation -- ovn-northd -- dp-groups=no --
> parallelization=yes ok                   
> 236: IPv6 prefix delegation -- ovn-northd -- dp-groups=no --
> parallelization=no ok                    
> 237: IPv6 prefix delegation -- ovn-northd-ddlog -- dp-groups=yes --
> parallelization=yes skipped (system-ovn.at:5197)
> 238: IPv6 prefix delegation -- ovn-northd-ddlog -- dp-groups=yes --
> parallelization=no skipped (system-ovn.at:5197)
> 239: IPv6 prefix delegation -- ovn-northd-ddlog -- dp-groups=no --
> parallelization=yes skipped (system-ovn.at:5197)
> 240: IPv6 prefix delegation -- ovn-northd-ddlog -- dp-groups=no --
> parallelization=no skipped (system-ovn.at:5197)
> 
> ## ------------- ##
> ## Test results. ##                                                         
> 
> ## ------------- ##                                                         
> 
> 
> 4 tests were successful.
> 4 tests were skipped.

Hi Jianlin,

it usually took couple of tries to have a failure,
if you would run it in a loop it will fail eventually.
AFAIK there is not a better way to reproduce it 
as it mainly relies on the order of the list which is always 
random so there is a certain chance that it happens.
Comment 10 Jianlin Shi 2022-09-21 06:35:06 UTC 
tested with following steps:

1. download src.rpm
2. install the rpm
3. make rpm with " rpmbuild  -bb ovn22.06.spec "
4. run test in BUILD/ovn-22.06.0 with "while :; do if ! make check-kernel TESTSUITEFLAGS="233"; then break; fi; done"

result on ovn22.06-22.06-27.el8:

system-ovn.at:5197: waiting while test "$(ovn-nbctl get logical_router_port rp-public ipv6_prefix | cut -c4-15)" = ""...
system-ovn.at:5197: wait failed after 30 seconds
./ovs-macros.at:259: hard failure
2022-09-21T05:52:37Z|00001|daemon_unix|WARN|/root/rpmbuild/BUILD/ovn-22.06.0/tests/system-kmod-testsuite.dir/233/ovs-vswitchd.pid: open: No such file or directory
ovs-appctl: cannot read pidfile "/root/rpmbuild/BUILD/ovn-22.06.0/tests/system-kmod-testsuite.dir/233/ovs-vswitchd.pid" (No such file or directory)
ovs-dpctl: opening datapath (No such device)
233. system-ovn.at:5197: 233. IPv6 prefix delegation -- ovn-northd -- dp-groups=yes -- parallelization=yes (system-ovn.at:5197): FAILED (ovs-macros.at:259)

result on ovn22.06-22.06.0-57.el8:

system-ovn.at:5033: wait failed after 30 seconds
./ovs-macros.at:259: hard failure
2022-09-21T06:20:51Z|00001|daemon_unix|WARN|/root/rpmbuild/BUILD/ovn-22.06.0/tests/system-kmod-testsuite.dir/233/ovs-vswitchd.pid: open: No such file or directory
ovs-appctl: cannot read pidfile "/root/rpmbuild/BUILD/ovn-22.06.0/tests/system-kmod-testsuite.dir/233/ovs-vswitchd.pid" (No such file or directory)
ovs-dpctl: opening datapath (No such device)
233. system-ovn.at:5033: 233. ACL after lb - reject -- ovn-northd -- dp-groups=yes -- parallelization=yes (system-ovn.at:5033): FAILED (ovs-macros.at:259)

confirmed that the patch in in ovn22.06-22.06.0-57, set SanityOnly
Comment 12 errata-xmlrpc 2022-10-11 12:25:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn22.06 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:6871