Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 2100393

Summary: ovn binaries crashes with Illegal instruction (core dumped) on certain CPU models
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Yatin Karel <ykarel>
Component: openvswitch2.17Assignee: Ales Musil <amusil>
Status: CLOSED ERRATA QA Contact: Zhiqiang Fang <zfang>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: FDP 22.BCC: amusil, ctrautma, dhill, dmarchan, jhsiao, jiji, jishi, ralongi, zfang
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openvswitch2.17-2.17.0-28.el8fdp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2102618 (view as bug list) Environment:
Last Closed: 2022-09-06 18:49:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2102618    

Description Yatin Karel 2022-06-23 09:12:51 UTC
Description of problem:
It is originally detected in OpenStack Upstream CI https://bugs.launchpad.net/tripleo/+bug/1979276 with ovn-2021-21.12.0-46, issue was not seen with last FDP release ovn-2021-21.12.0-11 so it's a regression in new FDP 22.B release.

OVN binaries like ovn-northd, ovn-controller, ovn-nbctl, ovn-sbctl etc crashes with "Illegal instruction (core dumped)"

For example, ovn-nbctl --version crashed as below:-
# ovn-nbctl --version
Illegal instruction (core dumped)

# coredumpctl info
           PID: 640886 (ovn-nbctl)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 4 (ILL)
     Timestamp: Thu 2022-06-23 08:48:35 UTC (3s ago)
  Command Line: ovn-nbctl --version
    Executable: /usr/bin/ovn-nbctl
 Control Group: /machine.slice/libpod-449776acdb3089ad2f92d49b850b234089c5ec549b6f5d0fcfb5414b5f19717a.scope/container
          Unit: libpod-449776acdb3089ad2f92d49b850b234089c5ec549b6f5d0fcfb5414b5f19717a.scope
         Slice: machine.slice
       Boot ID: 4f2c55fc25f34c84a6160468479ece43
    Machine ID: c26d255f89064955aa655cf12e74d969
      Hostname: standalone.localdomain
       Storage: /var/lib/systemd/coredump/core.ovn-nbctl.0.4f2c55fc25f34c84a6160468479ece43.640886.1655974115000000.zst (present)
     Disk Size: 160.0K
       Message: Process 640886 (ovn-nbctl) of user 0 dumped core.
                
                Module /usr/bin/ovn-nbctl with build-id 2798d30ce0833d6e0fcabb6d8a0a98cba4da707d
                Module linux-vdso.so.1 with build-id 826a46efc5a1c4a55cc6fdceeb06554eda66067e
                Module libnghttp2.so.14 with build-id 7eadbd56a0e5bcd3d8a6b39b9bab2327e380283a
                Module libpython3.9.so.1.0 with build-id bb4578c381c6d22045835e803bf846e2b5a28502
                Module libevent-2.1.so.7 with build-id af406c254338ff6ceff47360cba92cdcf233cf14
                Module libprotobuf-c.so.1 with build-id 46661ae5d66cbaa2aa82b1b765472bdfa4712a24
                Module ld-linux-x86-64.so.2 with build-id 1d95aae3e4174446d3b885ad234d4f7e573e71db
                Module libz.so.1 with build-id 25486226566596e403da5485fb0ec85deed6b9fa
                Module libc.so.6 with build-id 14830f7e71953d5f0dac317543ac1e3fcdd874f5
                Module libunbound.so.8 with build-id def32d1bb7a7d99c59bf62e00c628af0246afa91
                Module libm.so.6 with build-id 3eb525d2e163793ef2e888d5bb46e104d11a3201
                Module libcap-ng.so.0 with build-id fdca0a301667e15db99d726152b57feeb35e4dbe
                Module libcrypto.so.3 with build-id 12bfb8486a63c1daa0d3b1d901401cd152c09f8e
                Module libssl.so.3 with build-id 4f82a7edeeafe3698ccc5442d011a8cd5aaf4e9d
                Stack trace of thread 96216:
                #0  0x000055d209c3dba8 n/a (/usr/bin/ovn-nbctl + 0x16ba8)
                ELF object binary architecture: AMD x86-64

- Seen issue with below cpu models
Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
Intel Xeon E312xx (Sandy Bridge)

- Not seen issue with below cpu models
Intel Core Processor (Haswell, no TSX)
Intel Xeon Processor (Cascadelake)
AMD EPYC-Rome Processor

/proc/cpuinfo looks like below on affected system:-
===========================
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
stepping : 7
microcode : 0x71a
cpu MHz : 2593.881
cache size : 20480 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush acpi mmx fxsr sse sse2 syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl cpuid tsc_known_freq pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm cpuid_fault pti ssbd ibrs ibpb stibp xsaveopt md_clear flush_l1d
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit
bogomips : 5187.52
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
===========================
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
stepping : 4
microcode : 0x42e
cpu MHz : 2599.955
cache size : 20480 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush acpi mmx fxsr sse sse2 syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl cpuid tsc_known_freq pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cpuid_fault pti ssbd ibrs ibpb stibp fsgsbase smep erms xsaveopt md_clear flush_l1d
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit
bogomips : 5200.03
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
================================================


Version-Release number of selected component (if applicable):
- ovn-2021-21.12.0-46

How reproducible:
Always on certain CPU models

Steps to Reproduce:
1. Install affected ovn version on vms with one of below cpu models:-
Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
Intel Xeon E312xx (Sandy Bridge)
2. Run ovn-nbctl --version


Actual results:
Fails with Illegal instruction (core dumped)

Expected results:
Should succeed

Additional info:

Comment 2 Ales Musil 2022-06-23 11:39:57 UTC
So the illegal instruction is "shlx" which is part of BMI2 and the affected CPU do not support that. 

The cpu.c is compiled with -mbmi2 flag and others that might not be really supported (avx512, bmi1, etc.):

libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I ./include -I ./include -I ./lib -I ./lib -mavx512f -mavx512bw -mavx512dq -mbmi -mbmi2 -fPIC -Wstrict-prototypes -Wall -Wextra -Wno-sign-compare -Wpointer-arith -Wformat -Wformat-security -Wswitch-enum -Wunused-parameter -Wbad-function-cast -Wcast-align -Wstrict-prototypes -Wold-style-definition -Wmissing-prototypes -Wmissing-field-initializers -fno-strict-aliasing -Wswitch-bool -Wlogical-not-parentheses -Wsizeof-array-argument -Wbool-compare -Wshift-negative-value -Wduplicated-cond -Wshadow -Wmultistatement-macros -Wcast-align=strict -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -c lib/cpu.c -o lib/libopenvswitchavx512_la-cpu.o


Which I am not sure if that really makes sense, as this should detect the capabilities so it shouldn't be compiled with assumptions about those capabilities.

Comment 5 Ales Musil 2022-06-23 12:29:44 UTC
Moving to openvswitch

Comment 8 David Marchand 2022-06-24 07:37:45 UTC
I posted a more complete fix, following upstream report of AVX512 breakage, and discussion with Ilya:
https://patchwork.ozlabs.org/project/openvswitch/patch/20220624072959.240183-1-david.marchand@redhat.com/

Comment 10 OvS team 2022-06-29 14:03:31 UTC
* Wed Jun 29 2022 Open vSwitch CI <ovs-ci> - 2.17.0-28
- Merging upstream branch-2.17 [RH git: f3aee3f437]
    Commit list:
    a77ad9693c dpif-netdev: Refactor AVX512 runtime checks. (#2100393)

Comment 11 David Hill 2022-06-29 15:40:13 UTC
ovs binaries too ... 

[root@undercloud-0-rhosp17 ~]# ovs-vsctl show
Illegal instruction (core dumped)

Comment 12 David Hill 2022-06-29 16:04:29 UTC
I got this issue with

2022-06-29T10:32:47-0400 SUBDEBUG Installed: openvswitch2.17-2.17.0-18.el9fdp.x86_64


flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cpuid_fault pti ibrs ibpb tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt arat


and installing the brew package -25 solved this issue.

Comment 13 Jianlin Shi 2022-07-13 07:23:43 UTC
tested with vm provided by Yatin Karel, reproduced on openvswitch2.17-23:

[zuul@centos-9-stream-rax-ord-0030375827 ovs2.17-23]$ ovs-vsctl --version                             
Illegal instruction (core dumped)                                                                     
[zuul@centos-9-stream-rax-ord-0030375827 ovs2.17-23]$ rpm -qa | grep openvswitch                      
openvswitch-selinux-extra-policy-1.0-31.el9s.noarch                                                   
network-scripts-openvswitch2.15-2.15.0-99.el9s.x86_64                                                 
centos-release-nfv-openvswitch-1-4.el9s.noarch                                                        
openvswitch2.17-2.17.0-23.el9fdp.x86_64 

Verified on openvswitch2.17-28:

[zuul@centos-9-stream-rax-ord-0030375827 ovs2.17-28]$ ovs-vsctl --version                             
ovs-vsctl (Open vSwitch) 2.17.3                                                                       
DB Schema 8.3.0                                                                                       
[zuul@centos-9-stream-rax-ord-0030375827 ovs2.17-28]$ rpm -qa | grep openvswitch                      
openvswitch-selinux-extra-policy-1.0-31.el9s.noarch                                                   
network-scripts-openvswitch2.15-2.15.0-99.el9s.x86_64                                                 
centos-release-nfv-openvswitch-1-4.el9s.noarch                                                        
openvswitch2.17-2.17.0-28.el9fdp.x86_64

Comment 18 errata-xmlrpc 2022-09-06 18:49:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (openvswitch2.17 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:6368