Bug 1932642
| Summary: | ovn-controller (ovn2.13-20.12.0-23) is crashing when handling a deleted port binding | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux Fast Datapath | Reporter: | Numan Siddique <nusiddiq> |
| Component: | ovn2.13 | Assignee: | Dumitru Ceara <dceara> |
| Status: | CLOSED ERRATA | QA Contact: | Jianlin Shi <jishi> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | FDP 20.H | CC: | ctrautma, dcbw, dceara, ffernand, fhallal, jiji, jishi, jmelvin, mmichels, pmannidi, ralongi, sputhenp |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | ovn2.13-20.12.0-100 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-05-20 19:28:16 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Patches to fix the IDL bug have been sent upstream for review: https://patchwork.ozlabs.org/project/openvswitch/list/?series=231626 V3 posted for review: https://patchwork.ozlabs.org/project/openvswitch/list/?series=233663&state=* V4 posted for review: https://patchwork.ozlabs.org/project/openvswitch/list/?series=235642&state=* Fix merged in OSV IDL library. Posted OVN patch to pick up the IDL fix: http://patchwork.ozlabs.org/project/ovn/list/?series=237126&state=* tested with following script:
enable_coredump()
{
ulimit -c unlimited
ulimit -s unlimited
sysctl -w fs.suid_dumpable=2
if ! sysctl kernel.core_pattern | grep systemd-coredump
then
sysctl -w kernel.core_pattern="|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e"
fi
rm -rf /var/lib/systemd/coredump/*
rm -rf /run/log/journal/*
rm -rf /var/log/journal/*
systemctl restart systemd-journald
}
enable_coredump
systemctl start openvswitch
systemctl start ovn-northd
ovn-nbctl set-connection ptcp:6641
ovn-sbctl set-connection ptcp:6642
ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.175.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.175.25 external_ids:ovn-monitor-all=true
systemctl restart ovn-controller
ovn-nbctl --wait=hv -- lr-add r -- lrp-add r rp 00:00:00:00:00:01 1.0.0.1/24
ovn-nbctl lr-del r
sleep 1
coredumpctl info
reproduced on ovn2.13-20.12.0-95.el8:
[root@wsfd-advnetlab21 bz1932642]# rpm -qa | grep -E "openvswitch|ovn2.13"
ovn2.13-20.12.0-95.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-28.el8fdp.noarch
ovn2.13-central-20.12.0-95.el8fdp.x86_64
openvswitch2.15-2.15.0-15.el8fdp.x86_64
python3-openvswitch2.15-2.15.0-15.el8fdp.x86_64
ovn2.13-host-20.12.0-95.el8fdp.x86_64
+ ovn-nbctl --wait=hv -- lr-add r -- lrp-add r rp 00:00:00:00:00:01 1.0.0.1/24
+ ovn-nbctl lr-del r
+ sleep 1
+ coredumpctl info
PID: 196107 (ovn-controller)
UID: 992 (openvswitch)
GID: 989 (openvswitch)
Signal: 11 (SEGV)
Timestamp: Wed 2021-04-28 04:47:32 EDT (1s ago)
Command Line: ovn-controller unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --user openvswitch:openvswitch --no-chdir --log-file=/var/log/ovn/ovn-controller.log --pidfile=/run/ov>
Executable: /usr/bin/ovn-controller
Control Group: /system.slice/ovn-controller.service
Unit: ovn-controller.service
Slice: system.slice
Boot ID: 8bb53645164748e69ae6375b2d7907e6
Machine ID: 5f536de6b8d34068b40fc43b1c80523f
Hostname: wsfd-advnetlab21.anl.lab.eng.bos.redhat.com
Storage: /var/lib/systemd/coredump/core.ovn-controller.992.8bb53645164748e69ae6375b2d7907e6.196107.1619599652000000.lz4
Message: Process 196107 (ovn-controller) of user 992 dumped core.
Stack trace of thread 196107:
#0 0x000055e82c29e567 handle_deleted_lport (ovn-controller)
#1 0x000055e82c2a16e6 binding_handle_port_binding_changes (ovn-controller)
#2 0x000055e82c2c5af8 runtime_data_sb_port_binding_handler (ovn-controller)
#3 0x000055e82c2e043b engine_run (ovn-controller)
#4 0x000055e82c29b56b main (ovn-controller)
#5 0x00007f5acd059803 __libc_start_main (libc.so.6)
#6 0x000055e82c29cc6e _start (ovn-controller)
Stack trace of thread 196111:
#0 0x000055e82c37be8b time_timespec__ (ovn-controller)
#1 0x000055e82c37c107 time_wall_msec (ovn-controller)
#2 0x000055e82c38707e format_log_message (ovn-controller)
#3 0x000055e82c3888c6 vlog_valist (ovn-controller)
#4 0x000055e82c3880f3 vlog (ovn-controller)
#5 0x000055e82c3068af fatal_signal_run (ovn-controller)
#6 0x000055e82c371b00 poll_block (ovn-controller)
#7 0x000055e82c370c38 stopwatch_thread (ovn-controller)
#8 0x000055e82c35d5e3 ovsthread_wrapper (ovn-controller)
#9 0x00007f5acdcc015a start_thread (libpthread.so.0)
#10 0x00007f5acd132f73 __clone (libc.so.6)
Stack trace of thread 196109:
#0 0x00007f5acd127cf1 __poll (libc.so.6)
#1 0x000055e82c37c4a5 time_poll (ovn-controller)
#2 0x000055e82c371abc poll_block (ovn-controller)
#3 0x000055e82c35afda ovsrcu_postpone_thread (ovn-controller)
#4 0x000055e82c35d5e3 ovsthread_wrapper (ovn-controller)
#5 0x00007f5acdcc015a start_thread (libpthread.so.0)
#6 0x00007f5acd132f73 __clone (libc.so.6)
Stack trace of thread 196108:
#0 0x00007f5acd127cf1 __poll (libc.so.6)
#1 0x000055e82c37c4a5 time_poll (ovn-controller)
#2 0x000055e82c371abc poll_block (ovn-controller)
#3 0x000055e82c2bcd36 pinctrl_handler (ovn-controller)
#4 0x000055e82c35d5e3 ovsthread_wrapper (ovn-controller)
#5 0x00007f5acdcc015a start_thread (libpthread.so.0)
#6 0x00007f5acd132f73 __clone (libc.so.6)
Verified on ovn2.13-20.12.0-104.el8:
[root@wsfd-advnetlab21 bz1932642]# rpm -qa | grep ovn2.13
ovn2.13-central-20.12.0-104.el8fdp.x86_64
ovn2.13-20.12.0-104.el8fdp.x86_64
ovn2.13-host-20.12.0-104.el8fdp.x86_64
+ ovn-nbctl --wait=hv -- lr-add r -- lrp-add r rp 00:00:00:00:00:01 1.0.0.1/24
+ ovn-nbctl lr-del r
+ sleep 1
+ coredumpctl info
No coredumps found.
tested with following script:
enable_coredump()
{
ulimit -c unlimited
ulimit -s unlimited
sysctl -w fs.suid_dumpable=2
if ! sysctl kernel.core_pattern | grep systemd-coredump
then
sysctl -w kernel.core_pattern="|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e"
fi
rm -rf /var/lib/systemd/coredump/*
rm -rf /run/log/journal/*
rm -rf /var/log/journal/*
systemctl restart systemd-journald
}
enable_coredump
systemctl start openvswitch
systemctl start ovn-northd
ovn-nbctl set-connection ptcp:6641
ovn-sbctl set-connection ptcp:6642
ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.175.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.175.25 external_ids:ovn-monitor-all=true
systemctl restart ovn-controller
ovn-nbctl --wait=hv -- lr-add r -- lrp-add r rp 00:00:00:00:00:01 1.0.0.1/24
ovn-nbctl lr-del r
sleep 1
coredumpctl info
reproduced on ovn2.13-20.12.0-95.el8:
[root@wsfd-advnetlab21 bz1932642]# rpm -qa | grep -E "openvswitch|ovn2.13"
ovn2.13-20.12.0-95.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-28.el8fdp.noarch
ovn2.13-central-20.12.0-95.el8fdp.x86_64
openvswitch2.15-2.15.0-15.el8fdp.x86_64
python3-openvswitch2.15-2.15.0-15.el8fdp.x86_64
ovn2.13-host-20.12.0-95.el8fdp.x86_64
+ ovn-nbctl --wait=hv -- lr-add r -- lrp-add r rp 00:00:00:00:00:01 1.0.0.1/24
+ ovn-nbctl lr-del r
+ sleep 1
+ coredumpctl info
PID: 196107 (ovn-controller)
UID: 992 (openvswitch)
GID: 989 (openvswitch)
Signal: 11 (SEGV)
Timestamp: Wed 2021-04-28 04:47:32 EDT (1s ago)
Command Line: ovn-controller unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --user openvswitch:openvswitch --no-chdir --log-file=/var/log/ovn/ovn-controller.log --pidfile=/run/ov>
Executable: /usr/bin/ovn-controller
Control Group: /system.slice/ovn-controller.service
Unit: ovn-controller.service
Slice: system.slice
Boot ID: 8bb53645164748e69ae6375b2d7907e6
Machine ID: 5f536de6b8d34068b40fc43b1c80523f
Hostname: wsfd-advnetlab21.anl.lab.eng.bos.redhat.com
Storage: /var/lib/systemd/coredump/core.ovn-controller.992.8bb53645164748e69ae6375b2d7907e6.196107.1619599652000000.lz4
Message: Process 196107 (ovn-controller) of user 992 dumped core.
Stack trace of thread 196107:
#0 0x000055e82c29e567 handle_deleted_lport (ovn-controller)
#1 0x000055e82c2a16e6 binding_handle_port_binding_changes (ovn-controller)
#2 0x000055e82c2c5af8 runtime_data_sb_port_binding_handler (ovn-controller)
#3 0x000055e82c2e043b engine_run (ovn-controller)
#4 0x000055e82c29b56b main (ovn-controller)
#5 0x00007f5acd059803 __libc_start_main (libc.so.6)
#6 0x000055e82c29cc6e _start (ovn-controller)
Stack trace of thread 196111:
#0 0x000055e82c37be8b time_timespec__ (ovn-controller)
#1 0x000055e82c37c107 time_wall_msec (ovn-controller)
#2 0x000055e82c38707e format_log_message (ovn-controller)
#3 0x000055e82c3888c6 vlog_valist (ovn-controller)
#4 0x000055e82c3880f3 vlog (ovn-controller)
#5 0x000055e82c3068af fatal_signal_run (ovn-controller)
#6 0x000055e82c371b00 poll_block (ovn-controller)
#7 0x000055e82c370c38 stopwatch_thread (ovn-controller)
#8 0x000055e82c35d5e3 ovsthread_wrapper (ovn-controller)
#9 0x00007f5acdcc015a start_thread (libpthread.so.0)
#10 0x00007f5acd132f73 __clone (libc.so.6)
Stack trace of thread 196109:
#0 0x00007f5acd127cf1 __poll (libc.so.6)
#1 0x000055e82c37c4a5 time_poll (ovn-controller)
#2 0x000055e82c371abc poll_block (ovn-controller)
#3 0x000055e82c35afda ovsrcu_postpone_thread (ovn-controller)
#4 0x000055e82c35d5e3 ovsthread_wrapper (ovn-controller)
#5 0x00007f5acdcc015a start_thread (libpthread.so.0)
#6 0x00007f5acd132f73 __clone (libc.so.6)
Stack trace of thread 196108:
#0 0x00007f5acd127cf1 __poll (libc.so.6)
#1 0x000055e82c37c4a5 time_poll (ovn-controller)
#2 0x000055e82c371abc poll_block (ovn-controller)
#3 0x000055e82c2bcd36 pinctrl_handler (ovn-controller)
#4 0x000055e82c35d5e3 ovsthread_wrapper (ovn-controller)
#5 0x00007f5acdcc015a start_thread (libpthread.so.0)
#6 0x00007f5acd132f73 __clone (libc.so.6)
Verified on ovn2.13-20.12.0-104.el8:
[root@wsfd-advnetlab21 bz1932642]# rpm -qa | grep ovn2.13
ovn2.13-central-20.12.0-104.el8fdp.x86_64
ovn2.13-20.12.0-104.el8fdp.x86_64
ovn2.13-host-20.12.0-104.el8fdp.x86_64
+ ovn-nbctl --wait=hv -- lr-add r -- lrp-add r rp 00:00:00:00:00:01 1.0.0.1/24
+ ovn-nbctl lr-del r
+ sleep 1
+ coredumpctl info
No coredumps found.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2080 |
Description of problem: ovn-controller (ovn2.13-20.12.0-23) is crashing when handling a deleted port binding. The back trace is warning: Loadable section ".note.gnu.property" outside of ELF segments Core was generated by `/usr/bin/ovn-controller --pidfile --log-file unix:/run/openvswitch/db.sock'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00005564c7f3f247 in handle_deleted_lport (pb=0x55655b1af050, b_ctx_in=0x7ffd6ba50a10, b_ctx_out=0x7ffd6ba509c0) at controller/binding.c:2068 2068 struct local_datapath *ld = [Current thread is 1 (Thread 0x7f2f41f8de00 (LWP 7))] Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-101.el8.x86_64 libcap-ng-0.7.9-5.el8.x86_64 libevent-2.1.8-5.el8.x86_64 openssl-libs-1.1.1c-16.el8_2.x86_64 python3-libs-3.6.8-23.el8.x86_64 unbound-libs-1.7.3-11.el8_2.x86_64 zlib-1.2.11-16.el8_2.x86_64 (gdb) bt #0 0x00005564c7f3f247 in handle_deleted_lport (pb=0x55655b1af050, b_ctx_in=0x7ffd6ba50a10, b_ctx_out=0x7ffd6ba509c0) at controller/binding.c:2068 #1 0x00005564c7f423c6 in binding_handle_port_binding_changes (b_ctx_in=b_ctx_in@entry=0x7ffd6ba50a10, b_ctx_out=b_ctx_out@entry=0x7ffd6ba509c0) at controller/binding.c:2300 #2 0x00005564c7f65ad8 in runtime_data_sb_port_binding_handler (node=0x7ffd6ba51d80, data=0x5564c8c21240) at controller/ovn-controller.c:1507 #3 0x00005564c7f7f783 in engine_compute (recompute_allowed=<optimized out>, node=<optimized out>) at lib/inc-proc-eng.c:306 #4 engine_run_node (recompute_allowed=false, node=0x7ffd6ba51d80) at lib/inc-proc-eng.c:352 #5 engine_run (recompute_allowed=recompute_allowed@entry=false) at lib/inc-proc-eng.c:377 #6 0x00005564c7f3c9f6 in main (argc=<optimized out>, argv=<optimized out>) at controller/ovn-controller.c:2824 (gdb) frame 0 #0 0x00005564c7f3f247 in handle_deleted_lport (pb=0x55655b1af050, b_ctx_in=0x7ffd6ba50a10, b_ctx_out=0x7ffd6ba509c0) at controller/binding.c:2068 2068 struct local_datapath *ld = (gdb) ping ld Undefined command: "ping". Try "help". (gdb) print ld $1 = <optimized out> (gdb) print [b A syntax error in expression, near `[b'. (gdb) print pb $2 = (const struct sbrec_port_binding *) 0x55655b1af050 (gdb) print pb->logical_port $3 = 0x5565489a1f50 "lrp-41bb1c69-2387-4fd5-b1e5-a5a1785adcae" (gdb) print *pb->logical_port $4 = 108 'l' (gdb) print *pb $5 = {header_ = {hmap_node = {hash = 2636451573, next = 0x5564fa7831b0}, uuid = {parts = {2636451573, 2035437159, 2930629353, 391919587}}, src_arcs = {prev = 0x55655b1af070, next = 0x55655b1af070}, dst_arcs = {prev = 0x55655b1af080, next = 0x55655b1af080}, table = 0x5564c8cad4a8, old_datum = 0x0, parsed = true, new_datum = 0x0, prereqs = 0x0, written = 0x0, txn_node = {hash = 0, next = 0x1}, map_op_written = 0x0, map_op_lists = 0x0, set_op_written = 0x0, set_op_lists = 0x0, change_seqno = {0, 0, 2243844}, track_node = {prev = 0x5564c8cad528, next = 0x5565159a3b60}, updated = 0x0, tracked_old_datum = 0x55655b1af200}, chassis = 0x0, datapath = 0x0, encap = 0x0, external_ids = {map = { buckets = 0x55655b1af140, one = 0x0, mask = 0, n = 0}}, gateway_chassis = 0x0, n_gateway_chassis = 0, ha_chassis_group = 0x0, logical_port = 0x5565489a1f50 "lrp-41bb1c69-2387-4fd5-b1e5-a5a1785adcae", mac = 0x55652d1f7ae0, n_mac = 1, nat_addresses = 0x0, n_nat_addresses = 0, options = {map = {buckets = 0x55655b1af1a0, one = 0x0, mask = 0, n = 0}}, parent_port = 0x0, tag = 0x0, n_tag = 0, tunnel_key = 3, type = 0x5564e2e07350 "patch", \\ Looks like the tracked port_binding's datapath is NULL. Ideally this should not happen. The issue is seen in the Red Hat Training deployment Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: