Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
We have gone through a complete git bisect (plus scratch RPMs, container rebuilds and MetalLB tests) and found that commit "861590b24 bfdd: fix crash on show bfd peers counters json" is the first good one:
df7ab485b (tag: frr-7.5.1) FRRouting Release 7.5.1
(... 80 other bug fix commits ...)
46a2b560f tools: fix frr-reload BFD profile support [cherry-picked in frr-7.5-4.el8_4.1]
(... 110 other bug fix commits ...)
861590b24 bfdd: fix crash on show bfd peers counters json
(... 16 other bug fix commits ...)
35e42b176 (tag: frr-7.5) FRRouting Release 7.5
I built a scratch RPM cherry-picking commit "861590b24 bfdd: fix crash on show bfd peers counters json" and ran the MetalLB BGP and BFD tests. All relevant tests passed successfully.
We believe it would be easier and safer for RHEL to bump the RPM to 7.5.1 rather than risking instability by cherry-picking multiple patches. I understand package rebases are not usually carried in z-streams.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (frr bug fix and enhancement update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2022:1778
Description of problem: The bfd configuration is not applied and bfdd crashes. The configuration taken from a CI run is ['router bgp 64512\n neighbor 192.168.111.1 port 179\n', 'bfd\n', 'bfd\n profile bar\n', 'router bgp 64512\n', 'router bgp 64512\n no bgp ebgp-requires-policy\n', 'router bgp 64512\n no bgp network import-check\n', 'router bgp 64512\n no bgp default ipv4-unicast\n', 'router bgp 64512\n bgp router-id 10.10.10.0\n', 'router bgp 64512\n neighbor 192.168.111.1 remote-as 64512\n', 'router bgp 64512\n neighbor 192.168.111.1 port 179\n', 'router bgp 64512\n neighbor 192.168.111.1 timers 30 90\n', 'router bgp 64512\n neighbor 192.168.111.1 password ibgp-test\n', 'router bgp 64512\n neighbor 192.168.111.1 bfd profile bar\n', 'router bgp 64512\n address-family ipv4 unicast\n', 'router bgp 64512\n' ' address-family ipv4 unicast\n' ' neighbor 192.168.111.1 activate\n', 'router bgp 64512\n' ' address-family ipv4 unicast\n' ' neighbor 192.168.111.1 route-map 192.168.111.1-in in\n', 'router bgp 64512\n address-family ipv6 unicast\n', 'router bgp 64512\n' ' address-family ipv6 unicast\n' ' neighbor 192.168.111.1 activate\n', 'router bgp 64512\n' ' address-family ipv6 unicast\n' ' neighbor 192.168.111.1 route-map 192.168.111.1-in in\n', 'bfd\n', 'bfd\n profile bar\n'] Version-Release number of selected component (if applicable): frr x86_64 7.5-4.el8_4.1 How reproducible: Very very often. We had sporadic test runs where a test passed. Steps to Reproduce: 1. 2. 3. From the logs of the container that launches the frr reloader: Checking the configuration file syntax 2021-12-17 09:52:07,528 INFO: Called via "Namespace(bindir='/usr/bin', confdir='/etc/frr', daemon='', debug=False, filename='/etc/frr_reloader/frr.conf', input=None, log_level='info', overwrite=False, pathspace=None, reload=False, rundir='/var/run/frr', stdout=True, test=True, vty_socket=None)" 2021-12-17 09:52:07,528 INFO: Loading Config object from file /etc/frr_reloader/frr.conf 2021-12-17 09:52:07,646 INFO: Loading Config object from vtysh show running 2021-12-17 09:52:07,786 INFO: "frr version 7.5" cannot be removed 2021-12-17 09:52:07,786 INFO: "frr defaults traditional" cannot be removed 2021-12-17 09:52:07,786 INFO: "service integrated-vtysh-config" cannot be removed 2021-12-17 09:52:07,786 INFO: "line vty" cannot be removed Lines To Delete =============== router bgp 64512 address-family ipv6 unicast no neighbor 192.168.111.1 activate router bgp 64512 address-family ipv6 unicast no neighbor 192.168.111.1 route-map 192.168.111.1-in in Lines To Add ============ router bgp 64512 neighbor 192.168.111.1 port 179 router bgp 64512 address-family ipv4 unicast network 192.168.10.0/32 bfd bfd profile bar Applying the configuration file 2021-12-17 09:52:07,851 INFO: Called via "Namespace(bindir='/usr/bin', confdir='/etc/frr', daemon='', debug=False, filename='/etc/frr_reloader/frr.conf', input=None, log_level='info', overwrite=True, pathspace=None, reload=True, rundir='/var/run/frr', stdout=True, test=False, vty_socket=None)" 2021-12-17 09:52:07,851 INFO: Loading Config object from file /etc/frr_reloader/frr.conf 2021-12-17 09:52:08,084 INFO: Loading Config object from vtysh show running 2021-12-17 09:52:08,236 INFO: "frr version 7.5" cannot be removed 2021-12-17 09:52:08,236 INFO: "frr defaults traditional" cannot be removed 2021-12-17 09:52:08,236 INFO: "service integrated-vtysh-config" cannot be removed 2021-12-17 09:52:08,236 INFO: "line vty" cannot be removed 2021-12-17 09:52:08,350 INFO: Executed "router bgp 64512 address-family ipv6 unicast no neighbor 192.168.111.1 activate" 2021-12-17 09:52:08,479 INFO: Executed "router bgp 64512 address-family ipv6 unicast no neighbor 192.168.111.1 route-map 192.168.111.1-in in" 2021-12-17 09:52:08,480 INFO: /var/run/frr/reload-SVZK8L.txt content ['router bgp 64512\n neighbor 192.168.111.1 port 179\n', 'router bgp 64512\n address-family ipv4 unicast\n network 192.168.10.0/32 \n', 'bfd\n', 'bfd\n profile bar\n'] 2021-12-17 09:52:08,593 INFO: Loading Config object from vtysh show running 2021-12-17 09:52:08,729 INFO: "frr version 7.5" cannot be removed 2021-12-17 09:52:08,729 INFO: "frr defaults traditional" cannot be removed 2021-12-17 09:52:08,729 INFO: "service integrated-vtysh-config" cannot be removed 2021-12-17 09:52:08,729 INFO: "line vty" cannot be removed 2021-12-17 09:52:08,729 INFO: /var/run/frr/reload-ED9NJ8.txt content ['router bgp 64512\n neighbor 192.168.111.1 port 179\n', 'bfd\n', 'bfd\n profile bar\n', 'router bgp 64512\n neighbor 192.168.111.1 port 179\n', 'router bgp 64512\n address-family ipv4 unicast\n network 192.168.10.0/32 \n', 'bfd\n', 'bfd\n profile bar\n'] Traceback (most recent call last): File "/usr/lib/frr/frr-reload.py", line 1543, in <module> vtysh('write') File "/usr/lib/frr/frr-reload.py", line 106, in __call__ % (proc.returncode, command)) __main__.VtyshException: vtysh returned status 1 for command "write" Failed to fully apply configuration file From the logs of frr (full logs at https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_frr/13/pull-ci-openshift-frr-master-metallb-e2e-metal-ipi-ovn/1471762899738300416/artifacts/metallb-e2e-metal-ipi-ovn/gather-extra/artifacts/pods/metallb-system_speaker-4qhd6_frr.log): 2021/12/17 09:50:53.415 BGP: bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 192.168.111.1 in vrf default BFD: Received signal 11 at 1639734653 (si_addr 0x5, PC 0x7ffaa44d1af2); aborting... BFD: /usr/lib64/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x5b) [0x7ffaa64ca26b] BFD: /usr/lib64/frr/libfrr.so.0(zlog_signal+0xe1) [0x7ffaa64ca461] BFD: /usr/lib64/frr/libfrr.so.0(+0x83af8) [0x7ffaa64eeaf8] BFD: /lib64/libpthread.so.0(+0x12b20) [0x7ffaa4750b20] BFD: /lib64/libc.so.6(+0x158af2) [0x7ffaa44d1af2] BFD: /usr/lib/frr/bfdd(+0x11d57) [0x564a226c2d57] BFD: /usr/lib64/frr/libfrr.so.0(hash_iterate+0x4a) [0x7ffaa64be9ba] BFD: /usr/lib/frr/bfdd(+0x10b02) [0x564a226c1b02] BFD: /usr/lib64/frr/libfrr.so.0(+0x394f5) [0x7ffaa64a44f5] BFD: /usr/lib64/frr/libfrr.so.0(cmd_execute_command+0x71) [0x7ffaa64a66c1] BFD: /usr/lib64/frr/libfrr.so.0(cmd_execute+0xd0) [0x7ffaa64a68e0] BFD: /usr/lib64/frr/libfrr.so.0(+0x98d75) [0x7ffaa6503d75] BFD: /usr/lib64/frr/libfrr.so.0(+0x98f50) [0x7ffaa6503f50] BFD: /usr/lib64/frr/libfrr.so.0(+0x9b990) [0x7ffaa6506990] BFD: /usr/lib64/frr/libfrr.so.0(thread_call+0x5a) [0x7ffaa64fe27a] BFD: /usr/lib64/frr/libfrr.so.0(frr_run+0xe8) [0x7ffaa64c8de8] BFD: /usr/lib/frr/bfdd(main+0x27b) [0x564a226bb3ab] BFD: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7ffaa439c493] BFD: /usr/lib/frr/bfdd(_start+0x2e) [0x564a226bb58e] BFD: in thread vtysh_read scheduled from lib/vty.c:2682