Bug 2193145
| Summary: | Routes not recovered after frr restart for a long time | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Eduardo Olivares <eolivare> | |
| Component: | frr | Assignee: | Michal Ruprich <mruprich> | |
| Status: | CLOSED MIGRATED | QA Contact: | FrantiĊĦek Hrdina <fhrdina> | |
| Severity: | high | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 9.2 | CC: | fhrdina, lmiccini | |
| Target Milestone: | rc | Keywords: | MigratedToJIRA | |
| Target Release: | --- | Flags: | pm-rhel:
mirror+
|
|
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2218291 (view as bug list) | Environment: | ||
| Last Closed: | 2023-09-21 19:06:23 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 2218291 | |||
Thanks Eduardo for the report and for trying to identify fixing commit. Unfortunately, we already have this commit in our code: https://pkgs.devel.redhat.com/cgit/rpms/frr/commit/?h=rhel-9-main&id=9d939262e3a1057c264bb642137d970f48730875 I've added the patch 0006-graceful-restart together with the rebase. But nevermind that, at least we know that the latest version works and I can go on from there. I will get back to you as soon as I know more. Regards, Michal Hi Eduardo, I would like to ask you for the config on on of the nodes, maybe you forgot to add it here? Are you using graceful restart? Without it the change would not have been able to be transferred to the neighbor so quickly. frr-8.3.1-5 works for me when using graceful restart. The route disappears for a second but than it is added back. Maybe something in the config could help. Thanks, Michal (In reply to Michal Ruprich from comment #2) > Hi Eduardo, > > I would like to ask you for the config on on of the nodes, maybe you forgot > to add it here? Are you using graceful restart? Without it the change would > not have been able to be transferred to the neighbor so quickly. frr-8.3.1-5 > works for me when using graceful restart. The route disappears for a second > but than it is added back. Maybe something in the config could help. > > Thanks, > Michal Apologies for the delayed reply, Michal. I repeated the test. Same nodes involved and same frr version with identical result (leaf-0 and cmp-1-1, see comment#0). You can find frr logs and running configuration here: http://file.mad.redhat.com/~eolivare/BZ2193145/ 1) First I stopped frr on cmp-1-1, slept 1 sec and started frr. This happened between 09:41:43 and 09:41:48 (times are synchronized in both nodes). You can see the following logs on cmp-1-1_frr.log: 2023/06/05 09:41:53.071 BGP: [ZWCSR-M7FG9] enp2s0 [FSM] BGP_Start (Idle->Connect), fd -1 2023/06/05 09:41:53.071 BGP: [ZWCSR-M7FG9] enp3s0 [FSM] BGP_Start (Idle->Connect), fd -1 After this, the route to 172.30.1.3 (which is an IP on the cmp-1-1's loopback interface) was immediately recovered on leaf-0. 2) Later I stopped frr on cmp-1-1 and started it again immediately. This happened at 09:43:12 You can see the following logs on cmp-1-1_frr.log: 2023/06/05 09:43:16.785 BGP: [ZWCSR-M7FG9] enp2s0 [FSM] BGP_Start (Idle->Connect), fd -1 2023/06/05 09:43:16.785 BGP: [ZWCSR-M7FG9] enp3s0 [FSM] BGP_Start (Idle->Connect), fd -1 After this, the route to 172.30.1.3 has not been recovered on leaf-0 after ~30 min. Adding some more info. BFD sessions are successfully established between leaf-0 and cmp-1-1.
[root@leaf-0 ~]# vtysh -c 'show bfd peer 100.65.1.10'
BFD Peer:
peer 100.65.1.10 local-address 100.65.1.9 vrf default interface eth5
ID: 3474991286
Remote ID: 788172463
Active mode
Status: up
Uptime: 43 minute(s), 0 second(s)
Diagnostics: ok
Remote diagnostics: ok
Peer Type: dynamic
Local timers:
Detect-multiplier: 10
Receive interval: 500ms
Transmission interval: 500ms
Echo receive interval: 50ms
Echo transmission interval: disabled
Remote timers:
Detect-multiplier: 10
Receive interval: 500ms
Transmission interval: 500ms
Echo receive interval: 50ms
[root@cmp-1-1 /]# vtysh -c "show bfd peer 100.65.1.9"
BFD Peer:
peer 100.65.1.9 local-address 100.65.1.10 vrf default interface enp2s0
ID: 788172463
Remote ID: 3474991286
Active mode
Status: up
Uptime: 45 minute(s), 10 second(s)
Diagnostics: ok
Remote diagnostics: ok
Peer Type: dynamic
Local timers:
Detect-multiplier: 10
Receive interval: 500ms
Transmission interval: 500ms
Echo receive interval: 50ms
Echo transmission interval: disabled
Remote timers:
Detect-multiplier: 10
Receive interval: 500ms
Transmission interval: 500ms
Echo receive interval: 50ms
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug. This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there. Due to differences in account names between systems, some fields were not replicated. Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information. To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer. You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like: "Bugzilla Bug" = 1234567 In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information. |
Description of problem: FRR is running on two peer nodes (frr-8.3.1-5.el9.x86_64). One of them exposes via BGP a route to an IP from its loopback interface: [root@cmp-1-1 ~]# ip a s lo 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet 172.30.1.3/32 brd 172.30.1.3 scope host lo valid_lft forever preferred_lft forever The frr config running on this node can be found attached. The peer node, leaf-0, receives a route to 172.30.1.3 via BGP (100.65.1.10 is another IP on a physical interface at cmp-1-1): [root@leaf-0 ~]# vtysh -c 'show ip route 172.30.1.3' Routing entry for 172.30.1.3/32 Known via "bgp", distance 200, metric 0, best Last update 00:01:14 ago * 100.65.1.10, via eth5, weight 1 When frr is restarted on cmp-1-1, the route to 172.30.1.3 is removed from leaf-0 and it takes several minutes to recover it, more than 5 minutes, which is not acceptable. When frr is stopped on cmp-1-1, wait 1 second, and started again, the route to 172.30.1.3 is removed from leaf-0, but it is recovered almost immediately, which is fine. The frr version frr-8.3.1-5.el9.x86_64 has been updated on leaf-0 to frr-8.5.1-02.el9.x86_64 (it has been obtained from [1]). The test is repeated and now the routes are recovered immediately after restarting frr on cmp-1-1. We believe this is the PR that fixes this issue: https://github.com/FRRouting/frr/pull/12034 Can this patch be backported to RHEL9.2 frr version? (or can it be rebased? whatever is usually done in these cases). [1] https://rpm.frrouting.org/ Version-Release number of selected component (if applicable): frr-8.3.1-5.el9.x86_64 How reproducible: 100% Steps to Reproduce: 1. restart frr on one node 2. check the BGP routes on its peer node 3. Actual results: BGP routes are not recovered after the frr restart Expected results: BGP routes should be recovered after the frr restart Additional info: