Bug 2104779
| Summary: | openvswitch coredump when work with ovn-2021-21.12.0-82.el8 | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux Fast Datapath | Reporter: | Jianlin Shi <jishi> |
| Component: | openvswitch2.15 | Assignee: | Mike Pattrick <mpattric> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Jianlin Shi <jishi> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | FDP 22.F | CC: | ctrautma, fleitner, hewang, jhsiao, mpattric, ralongi, zfang |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-11-24 16:41:23 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
*** Bug 2130930 has been marked as a duplicate of this bug. *** This does look to be the same issue as bz2130930, there are over 8000 stack frames in this dump. There appears to be an infinite loop. At table 65 there is a clone, a clear conntrack, and then a resubmit back to table 8. resubmits are limited to 4096, however, we hit due to clones and other forms of recursion we blow past the stack before hitting the resubmit limit. THere's also a "depth" limit of 64, but in this dump we crashed at 61. We could probably prevent a segfault by increasing the stack size - this core dump only appears to have a 2MB stack. By increasing the stack to 4MB we would hit an error condition instead of overrunning the stack. I have one bit of confusion about this issue. I see in "networking/openvswitch/ovn/common/include.sh" the stack limit is set to either 8192 or unlimited. However, in the core dump, the stack maximum seem to be set to 2048. With a larger stack maximum, this issue would be caught with xlate_resubmit_resource_check() and there would be no crash. Perhaps xlate_resubmit_resource_check() should dynamically detect the stack conditions as a safety measure, but I'm curious why the stack is so limited in this environment. Is there anything that would cause the stack size to be set to 2048? (In reply to Michael Pattrick from comment #4) > I have one bit of confusion about this issue. > > I see in "networking/openvswitch/ovn/common/include.sh" the stack limit is > set to either 8192 or unlimited. However, in the core dump, the stack > maximum seem to be set to 2048. > > With a larger stack maximum, this issue would be caught with > xlate_resubmit_resource_check() and there would be no crash. Perhaps > xlate_resubmit_resource_check() should dynamically detect the stack > conditions as a safety measure, but I'm curious why the stack is so limited > in this environment. > > Is there anything that would cause the stack size to be set to 2048? no, the stack size is still ulimited when run the case: core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 379914 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) unlimited cpu time (seconds, -t) unlimited max user processes (-u) 379914 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Ah I see now, the threads stack is crammed in between the stacks of other threads and cannot grow any more. The initial stack size in OVS is set very small to account for small devices like WRT based systems, but this obviously doesn't work well with the large OVN tables. I'll send out a patch to improve this behavior. As a workaround for now, if you change the ovn include.sh enable_coredump() to "ulimit -s 8192", this problem will probably go away, as the default buffer between stacks will be 8MB instead of 2MB. (In reply to Michael Pattrick from comment #6) > Ah I see now, the threads stack is crammed in between the stacks of other > threads and cannot grow any more. > > The initial stack size in OVS is set very small to account for small devices > like WRT based systems, but this obviously doesn't work well with the large > OVN tables. I'll send out a patch to improve this behavior. > > As a workaround for now, if you change the ovn include.sh enable_coredump() > to "ulimit -s 8192", this problem will probably go away, as the default > buffer between stacks will be 8MB instead of 2MB. after set stack size as 8192, it still crash: core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 379914 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 379914 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited PID: 44337 (ovs-vswitchd) UID: 993 (openvswitch) GID: 1001 (hugetlbfs) Signal: 11 (SEGV) Timestamp: Wed 2022-10-12 22:01:11 EDT (22min ago) Command Line: ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log > Executable: /usr/sbin/ovs-vswitchd Control Group: / Slice: -.slice Boot ID: 851f552751e44f428c042f674f53489a Machine ID: 45ea5dd99e914ba58f65688a572842af Hostname: wsfd-advnetlab16.anl.lab.eng.bos.redhat.com Storage: /var/lib/systemd/coredump/core.ovs-vswitchd.993.851f552751e44f428c042f674f53489a.44337.1665626471000000.lz4 Message: Process 44337 (ovs-vswitchd) of user 993 dumped core. Stack trace of thread 44362: #0 0x0000559563da72d1 classifier_lookup__ (ovs-vswitchd) #1 0x0000559563d6ed23 rule_dpif_lookup_from_table (ovs-vswitchd) #2 0x0000559563d87358 xlate_table_action (ovs-vswitchd) #3 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #4 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #5 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #6 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #7 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #8 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #9 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #10 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #11 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #12 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #13 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #14 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #15 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #16 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #17 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #18 0x0000559563d91061 clone_xlate_actions (ovs-vswitchd) #19 0x0000559563d8de0c do_xlate_actions (ovs-vswitchd) #20 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #21 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #22 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #23 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #24 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #25 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #26 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #27 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #28 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #29 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #30 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #31 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #32 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #33 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #34 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #35 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #36 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #37 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #38 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #39 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #40 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #41 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #42 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #43 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #44 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #45 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #46 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #47 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #48 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #49 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #50 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #51 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #52 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #53 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #54 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #55 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #56 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #57 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #58 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #59 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #60 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #61 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) #62 0x0000559563d872ce xlate_table_action (ovs-vswitchd) #63 0x0000559563d8e1bc do_xlate_actions (ovs-vswitchd) ...... the issue still exist when test with ovn22.03-22.03.0-218 + openvswitch3.1-3.1.0-45: https://beaker.engineering.redhat.com/recipes/14350215#task164016017 but the issue didn't occur when test with ovn23.03-23.03.0-86 + openvswitch3.1-3.1.0-45 (In reply to Jianlin Shi from comment #8) > the issue still exist when test with ovn22.03-22.03.0-218 + > openvswitch3.1-3.1.0-45: > https://beaker.engineering.redhat.com/recipes/14350215#task164016017 > > but the issue didn't occur when test with ovn23.03-23.03.0-86 + > openvswitch3.1-3.1.0-45 https://beaker.engineering.redhat.com/recipes/14222788#task162837716 Hello Jianlin, The patch related to this has been accepted upstream: https://github.com/openvswitch/ovs/commit/4829506b2a21ca42628ea7f73d8c4cf82cb11f9f However, this patch doesn't seem to have made it to the downstream repo yet. (In reply to Mike Pattrick from comment #10) Hi Michael, That patch was merged in master and in 3.2 (release candidate upstream), but it is not backported to stable branches like 3.1 or older. If you think this is a bug fix, we should request upstream to get it backported to older branches that makes sense as well. Please evaluate because we will need to fix 3.1 and 2.17 at least since this bug started with 2.15. fbl I've submitted backports for currently supported branches upstream. This patch has been backported |
Description of problem: openvswitch coredump when work with ovn-2021-21.12.0-82.el8. won't crash if work with ovn-2021-21.12.0-73.el8 Version-Release number of selected component (if applicable): openvswitch2.15-2.15.0-109.el8fdp.x86_64.rpm How reproducible: Always Steps to Reproduce: 1. run case ovn/nat:ovn_ipv4_nat 2. Actual results: PID: 433788 (ovs-vswitchd) UID: 988 (openvswitch) GID: 1001 (hugetlbfs) Signal: 11 (SEGV) Timestamp: Wed 2022-07-06 05:24:34 EDT (1min 6s ago) Command Line: ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach Executable: /usr/sbin/ovs-vswitchd Control Group: / Slice: -.slice Boot ID: 6d07c25be8c94bab9efb767e1568ae95 Machine ID: 8d05064f67c24e75adb507fe2e706e36 Hostname: wsfd-advnetlab18.anl.lab.eng.bos.redhat.com Storage: /var/lib/systemd/coredump/core.ovs-vswitchd.988.6d07c25be8c94bab9efb767e1568ae95.433788.1657099474000000.lz4 Message: Process 433788 (ovs-vswitchd) of user 988 dumped core. Stack trace of thread 433793: #0 0x000055d2b5b362d1 classifier_lookup__ (ovs-vswitchd) #1 0x000055d2b5afdd23 rule_dpif_lookup_from_table (ovs-vswitchd) #2 0x000055d2b5b16358 xlate_table_action (ovs-vswitchd) #3 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #4 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #5 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #6 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #7 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #8 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #9 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #10 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #11 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #12 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #13 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #14 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #15 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #16 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #17 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #18 0x000055d2b5b20061 clone_xlate_actions (ovs-vswitchd) #19 0x000055d2b5b1ce0c do_xlate_actions (ovs-vswitchd) #20 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #21 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #22 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #23 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #24 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #25 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #26 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #27 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #28 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #29 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #30 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #31 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #32 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #33 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #34 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #35 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #36 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #37 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #38 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #39 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #40 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #41 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #42 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #43 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #44 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #45 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #46 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #47 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #48 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #49 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #50 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #51 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #52 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #53 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #54 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #55 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #56 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #57 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #58 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #59 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #60 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #61 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) #62 0x000055d2b5b162ce xlate_table_action (ovs-vswitchd) #63 0x000055d2b5b1d1bc do_xlate_actions (ovs-vswitchd) Stack trace of thread 433804: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433801: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433810: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433788: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5474d0a main (ovs-vswitchd) #4 0x00007fcf805ddcf3 __libc_start_main (libc.so.6) #5 0x000055d2b5475f4e _start (ovs-vswitchd) Stack trace of thread 433796: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433800: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433799: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433795: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433797: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5bd9d8a ovsrcu_postpone_thread (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433812: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433794: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433813: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433809: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433798: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433814: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433807: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433818: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433817: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433808: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433802: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433803: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433811: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433805: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433815: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433816: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433820: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433806: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433823: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433822: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433828: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433827: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433832: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5bdc30c ovs_barrier_block (ovs-vswitchd) #4 0x000055d2b5b13069 udpif_revalidator (ovs-vswitchd) #5 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #6 0x00007fcf8265b1cf start_thread (libpthread.so.0) #7 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433837: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5bdc30c ovs_barrier_block (ovs-vswitchd) #4 0x000055d2b5b13069 udpif_revalidator (ovs-vswitchd) #5 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #6 0x00007fcf8265b1cf start_thread (libpthread.so.0) #7 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433836: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5bdc30c ovs_barrier_block (ovs-vswitchd) #4 0x000055d2b5b13069 udpif_revalidator (ovs-vswitchd) #5 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #6 0x00007fcf8265b1cf start_thread (libpthread.so.0) #7 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433835: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5bdc30c ovs_barrier_block (ovs-vswitchd) #4 0x000055d2b5b13069 udpif_revalidator (ovs-vswitchd) #5 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #6 0x00007fcf8265b1cf start_thread (libpthread.so.0) #7 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433830: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5bdc30c ovs_barrier_block (ovs-vswitchd) #4 0x000055d2b5b13069 udpif_revalidator (ovs-vswitchd) #5 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #6 0x00007fcf8265b1cf start_thread (libpthread.so.0) #7 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433824: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433825: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433841: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5bdc30c ovs_barrier_block (ovs-vswitchd) #4 0x000055d2b5b13069 udpif_revalidator (ovs-vswitchd) #5 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #6 0x00007fcf8265b1cf start_thread (libpthread.so.0) #7 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433926: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b08aa9 monitor_main (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433819: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433838: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5bdc30c ovs_barrier_block (ovs-vswitchd) #4 0x000055d2b5b13069 udpif_revalidator (ovs-vswitchd) #5 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #6 0x00007fcf8265b1cf start_thread (libpthread.so.0) #7 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433829: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b131a4 udpif_revalidator (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433826: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433839: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5bdc30c ovs_barrier_block (ovs-vswitchd) #4 0x000055d2b5b13069 udpif_revalidator (ovs-vswitchd) #5 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #6 0x00007fcf8265b1cf start_thread (libpthread.so.0) #7 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433840: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5bdc30c ovs_barrier_block (ovs-vswitchd) #4 0x000055d2b5b13069 udpif_revalidator (ovs-vswitchd) #5 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #6 0x00007fcf8265b1cf start_thread (libpthread.so.0) #7 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433821: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5b11d21 udpif_upcall_handler (ovs-vswitchd) #4 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #5 0x00007fcf8265b1cf start_thread (libpthread.so.0) #6 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433834: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5bdc30c ovs_barrier_block (ovs-vswitchd) #4 0x000055d2b5b13069 udpif_revalidator (ovs-vswitchd) #5 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #6 0x00007fcf8265b1cf start_thread (libpthread.so.0) #7 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433831: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5bdc30c ovs_barrier_block (ovs-vswitchd) #4 0x000055d2b5b13069 udpif_revalidator (ovs-vswitchd) #5 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #6 0x00007fcf8265b1cf start_thread (libpthread.so.0) #7 0x00007fcf805dcdd3 __clone (libc.so.6) Stack trace of thread 433833: #0 0x00007fcf806c7ac1 __poll (libc.so.6) #1 0x000055d2b5c07b65 time_poll (ovs-vswitchd) #2 0x000055d2b5bf09ac poll_block (ovs-vswitchd) #3 0x000055d2b5bdc30c ovs_barrier_block (ovs-vswitchd) #4 0x000055d2b5b13069 udpif_revalidator (ovs-vswitchd) #5 0x000055d2b5bdc403 ovsthread_wrapper (ovs-vswitchd) #6 0x00007fcf8265b1cf start_thread (libpthread.so.0) #7 0x00007fcf805dcdd3 __clone (libc.so.6) Expected results: no crash Additional info: the beaker job: https://beaker.engineering.redhat.com/recipes/12239237#task146990550 ovs doesn't crash if tested with: ovs + ovn-2021-21.12.0-73.el8