2032912 – Enabling xdp-tools and libxdp on architectures other than x86

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2032912 - Enabling xdp-tools and libxdp on architectures other than x86

Summary: Enabling xdp-tools and libxdp on architectures other than x86

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 9
Classification:	Red Hat
Component:	xdp-tools
Sub Component:
Version:	9.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Toke Høiland-Jørgensen
QA Contact:	Zhiqian Guan
Docs Contact:
URL:
Whiteboard:
Depends On:	2012854
Blocks:
TreeView+	depends on / blocked

Reported:	2021-12-15 13:14 UTC by Toke Høiland-Jørgensen
Modified:	2022-05-17 13:45 UTC (History)
CC List:	4 users (show)
Fixed In Version:	xdp-tools-1.2.3-1.el9
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2022-05-17 13:42:45 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	RHELPLAN-105888	0	None	None	None	2021-12-15 13:15:57 UTC
Red Hat Product Errata	RHEA-2022:2568	0	None	None	None	2022-05-17 13:42:48 UTC

Description Toke Høiland-Jørgensen 2021-12-15 13:14:07 UTC

In RHEL 8 we only ship xdp-tools for the x86 architecture because we can't support multiple programs on other architectures due to the lack of support in the kernel JIT for those architectures.

However, xdp-tools itself can run even without the kernel support and will just fall back to regular program attachment, so we'd like to start shipping xdp-tools for all architectures for RHEL 9.

The actual change is just a configuration change in the compose, so the timing depends on QE resources. Opening this bug to track that, with a tentative optimistic deadline of doing this for RHEL9.0...

Comment 1 Toke Høiland-Jørgensen 2022-01-05 13:31:10 UTC

Could we get an opinion from QE on whether this is feasible for 9.0, please? For the basic testing it should be enough to just run and verify the xdp-tools selftests on each architecture.

Comment 2 Ma Yuying 2022-01-06 01:44:06 UTC

Hi Zhiqian, will you please see comment#1 and give some feedback? Thanks!

Comment 3 Zhiqian Guan 2022-01-06 03:05:09 UTC

(In reply to Toke Høiland-Jørgensen from comment #1)
> Could we get an opinion from QE on whether this is feasible for 9.0, please?
> For the basic testing it should be enough to just run and verify the
> xdp-tools selftests on each architecture.

If we only need to run the selftests, I think we can do that on each arch. I'll try and let you know

Comment 4 Zhiqian Guan 2022-01-07 07:15:17 UTC

I managed to run xdp-tools self-testing on aarch64, I could pick a random non-x86 system in the beaker since it only needs a single machine to run selftest, so it's doable, but I need more testing times as random non-x86 system may not be available all the time

We just need to do the compose config change.

as for full testing for xdp-tools, we don't have such resources currently

Comment 5 Toke Høiland-Jørgensen 2022-01-13 23:27:46 UTC

Created MR for this here: https://gitlab.com/redhat/centos-stream/release-engineering/comps/-/merge_requests/177

Comment 8 Toke Høiland-Jørgensen 2022-01-14 13:21:23 UTC

Right, so the MR was merged; so this should be fixed once the updated build passes CI gating...

Comment 9 Zhiqian Guan 2022-01-21 06:19:59 UTC

postpone to ITM24 as I couldn't successfully run job because of the Beaker issue

Comment 10 Zhiqian Guan 2022-02-07 11:57:30 UTC

Hi Toke,  IIRC, the dispatcher is not supported in the non-x86 arch, so we can just ignore the failing test cases that using dispatcher, is that correct?

error:
  libxdp: Compatibility check for dispatcher program failed: Unknown error 524
  libxdp: Falling back to loading single prog without dispatcher

Comment 11 Toke Høiland-Jørgensen 2022-02-07 12:22:11 UTC

Yup, as long as the load succeeds after that error message; does that only show up when you're running with '-v' ?

Comment 12 Zhiqian Guan 2022-02-09 13:40:12 UTC

Hmm, I didn't use -v options when seeing the warning. And the case in different platforms is different, there're few issues now.

ISSUE ONE: 

on ppc64l: 
we can load the program, but cannot see it in xdp-loader status and also unload it via xdp-loader. we can only manage it by using "ip" cmd.
xdp-load with -vv option logging is attached

#### on ppc64l:
[root@ibm-p9z-25-lp6 xdp-tools]# xdp-loader load veth0 test_long_func_name.o
libxdp: Compatibility check for dispatcher program failed: Unknown error 524
libxdp: Falling back to loading single prog without dispatcher
[root@ibm-p9z-25-lp6 xdp-tools]# xdp-loader status
CURRENT XDP PROGRAM STATUS:

Interface        Prio  Program name      Mode     ID   Tag               Chain actions
--------------------------------------------------------------------------------------
lo                     <No XDP program loaded!>
env2                   <No XDP program loaded!>
veth1                  <No XDP program loaded!>
veth0                  <No XDP program loaded!>

[root@ibm-p9z-25-lp6 xdp-tools]# xdp-loader load veth0 test_long_func_name.o
libxdp: Compatibility check for dispatcher program failed: Unknown error 524
libxdp: Falling back to loading single prog without dispatcher
Couldn't attach XDP program on iface 'veth0': File exists(-17)
[root@ibm-p9z-25-lp6 xdp-tools]# xdp-loader status
CURRENT XDP PROGRAM STATUS:

Interface        Prio  Program name      Mode     ID   Tag               Chain actions
--------------------------------------------------------------------------------------
lo                     <No XDP program loaded!>
env2                   <No XDP program loaded!>
veth1                  <No XDP program loaded!>
veth0                  <No XDP program loaded!>

[root@ibm-p9z-25-lp6 xdp-tools]# ip link show veth0
4: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 46:d0:ed:73:15:5d brd ff:ff:ff:ff:ff:ff
    prog/xdp id 76 tag b5a46c6e9935298c jited
[root@ibm-p9z-25-lp6 xdp-tools]# xdp-loader unload veth0 -a
No XDP program loaded on veth0
[root@ibm-p9z-25-lp6 xdp-tools]# ip link show veth0
4: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 46:d0:ed:73:15:5d brd ff:ff:ff:ff:ff:ff
    prog/xdp id 76 tag b5a46c6e9935298c jited
[root@ibm-p9z-25-lp6 xdp-tools]#
[root@ibm-p9z-25-lp6 xdp-tools]# arch
ppc64le
[root@ibm-p9z-25-lp6 xdp-tools]#
[root@ibm-p9z-25-lp6 xdp-tools]# rpm -q xdp-tools
xdp-tools-1.2.1-1.el9.ppc64le
[root@ibm-p9z-25-lp6 xdp-tools]# uname -r
5.14.0-55.el9.ppc64le
[root@ibm-p9z-25-lp6 xdp-tools]#

Comment 14 Zhiqian Guan 2022-02-09 13:45:00 UTC

ISSUE TWO:

On ppc64le/s390x/aarch64:
We can use xdp-loader to load the program successfully but can also see warnings without using the -v option. 


[root@ibm-z-502 xdp-tools]# xdp-loader load veth0 test_long_func_name.o
libxdp: Compatibility check for dispatcher program failed: Unknown error 524
libxdp: Falling back to loading single prog without dispatcher
[root@ibm-z-502 xdp-tools]# xdp-loader status
CURRENT XDP PROGRAM STATUS:

Interface        Prio  Program name      Mode     ID   Tag               Chain actions
--------------------------------------------------------------------------------------
lo                     <No XDP program loaded!>
enc600                 <No XDP program loaded!>
veth1                  <No XDP program loaded!>
veth0                  xdp_test_prog_w   native   742  c33de9ac39ebcf23

[root@ibm-z-502 xdp-tools]# ip link show veth0
10: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether a2:61:a0:f2:df:92 brd ff:ff:ff:ff:ff:ff
    prog/xdp id 742 tag c33de9ac39ebcf23 jited
[root@ibm-z-502 xdp-tools]#

Comment 15 Zhiqian Guan 2022-02-09 14:00:41 UTC

ISSUE THREE:

on s390x/aarch64:
cases [test_capt_pcap][test_capt_pcapng] failed with below warning:

         ./test_runner.sh: line 140: kill: (-18823) - No such process
        ./test_runner.sh: line 142: kill: (-18823) - No such process
        tcpdump: truncated dump file; tried to read 4 file header bytes, only got 0
        ERROR: IPv6 packet not received

detailed logging attached

Comment 17 Zhiqian Guan 2022-02-09 14:03:50 UTC

ISSUE FOUR:

on aarch64/s390x:
[test_capt_term][test_exitentry][test_snap][test_promiscuous_preload] cases Failed with below warning:
         ./test_runner.sh: line 140: kill: (-18898) - No such process
        ./test_runner.sh: line 142: kill: (-18898) - No such process
        ERROR: Can't attach XDP trace fentry function: Unknown error 524
        ERROR: IPv6 packet not received

detailed logging attached

Comment 19 Zhiqian Guan 2022-02-09 14:08:29 UTC

ISSUE FIVE:

on aarch64/s390x:
[test_multi_pkt] [test_perf_wakeup] cases failed with below warning:
        ./test_runner.sh: line 140: kill: (-18983) - No such process
        ./test_runner.sh: line 142: kill: (-18983) - No such process

        ERROR: IPv6 packet not received for wakeup 0

     [test_multi_pkt]              FAIL
        libxdp: Compatibility check for dispatcher program failed: Unknown error 524
        libxdp: Falling back to loading single prog without dispatcher
        PING fc42:dead:cafe:1::2(fc42:dead:cafe:1::2) 56 data bytes

        --- fc42:dead:cafe:1::2 ping statistics ---
        20000 packets transmitted, 20000 received, 0% packet loss, time 131ms
        rtt min/avg/max/mdev = 0.002/0.003/0.054/0.000 ms, ipg/ewma 0.006/0.002 ms
        ./test_runner.sh: line 140: kill: (-18959) - No such process
        ./test_runner.sh: line 142: kill: (-18959) - No such process

        ERROR: IPv6 entry packet not received, 56
     [test_perf_wakeup]            FAIL
        libxdp: Compatibility check for dispatcher program failed: Unknown error 524
        libxdp: Falling back to loading single prog without dispatcher
        PING fc42:dead:cafe:1::2(fc42:dead:cafe:1::2) 56 data bytes
        64 bytes from fc42:dead:cafe:1::2: icmp_seq=1 ttl=64 time=0.059 ms

        --- fc42:dead:cafe:1::2 ping statistics ---
        1 packets transmitted, 1 received, 0% packet loss, time 0ms
        rtt min/avg/max/mdev = 0.059/0.059/0.059/0.000 ms
        ./test_runner.sh: line 140: kill: (-18983) - No such process
        ./test_runner.sh: line 142: kill: (-18983) - No such process

        ERROR: IPv6 packet not received for wakeup 0

Comment 20 Zhiqian Guan 2022-02-09 14:13:47 UTC

Hi Toke,

Please check the above 5 comments, there looks like some actual issues, I'll file bugs at least for the issue in comment12 and comment14, others may need your double confirm if they're caused by the same reason or different reasons, thanks!

Comment 21 Toke Høiland-Jørgensen 2022-02-10 17:35:53 UTC

Right, so the test failures are kinda expected: xdpdump relies on the same support as the multi-prog dispatcher so it's not going to work. We should probably detect this and skip the test (in fact I thought we already did this), but it's not critical, so I don't think you'll need to file bugs for them.

I'll comment on the other BZs for the two loader issues...

Comment 22 Zhiqian Guan 2022-02-28 17:11:38 UTC

I can now download xdp-tools on non-x86 arch system, xdpdump selftest failures are expected as per Comment21, set this to VERIFIED, unexpected warning issue is tracked by https://bugzilla.redhat.com/show_bug.cgi?id=2052546

[root@ibm-z-510 xdp-tools]# rpm -q xdp-tools
xdp-tools-1.2.1-1.el9.s390x
[root@ibm-z-510 xdp-tools]#

[root@hpe-apollo-cn99xx-14-vm-25 xdp-tools]# rpm -q xdp-tools
xdp-tools-1.2.3-1.el9.aarch64
[root@hpe-apollo-cn99xx-14-vm-25 xdp-tools]#

[root@ibm-p9z-07-lp4 xdp-tools]# rpm -q xdp-tools
xdp-tools-1.2.3-1.el9.ppc64le
[root@ibm-p9z-07-lp4 xdp-tools]#

Comment 25 errata-xmlrpc 2022-05-17 13:42:45 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: xdp-tools), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:2568

Note You need to log in before you can comment on or make changes to this bug.