Bug 1684477
| Summary: | Openvswich process is not restarted after a kill command, in 20% of the times | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux Fast Datapath | Reporter: | Timothy Redaelli <tredaelli> | |
| Component: | openvswitch2.11 | Assignee: | Timothy Redaelli <tredaelli> | |
| Status: | CLOSED ERRATA | QA Contact: | Rick Alongi <ralongi> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | FDP 19.03 | CC: | ctrautma, jhsiao, kfida, pvauter, ralongi, tli | |
| Target Milestone: | --- | |||
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | openvswitch2.11-2.11.0-2.el7fdn | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | 1653717 | |||
| : | 1684483 (view as bug list) | Environment: | ||
| Last Closed: | 2019-04-23 08:26:35 UTC | Type: | --- | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1684483 | |||
|
Description
Timothy Redaelli
2019-03-01 11:05:53 UTC
Created the script below to test the issue: [root@netqe20 ~]# cat bz1684477.sh #!/bin/bash iterations=${iterations:-"50"} result_file="/home/result_file.txt" rm -f $result_file check_final_result() { if [[ $(grep FAIL $result_file) ]]; then echo "$count crash iterations run of $iterations total" echo "Overall Test Result: FAIL" else let count-- echo "$count crash iterations run of $iterations total" echo "Overall Test Result: PASS" fi } test_systemctl_status_running() { service=$1 systemctl_status=$(systemctl status $service | grep Active | awk '{print $2$3}') if [[ $service == "openvswitch" ]]; then if [[ "$systemctl_status" == "active(exited)" ]]; then echo "PASS" >> $result_file else echo "FAIL" >> $result_file check_final_result exit 1 fi elif [[ $service == "ovs-vswitchd" ]] || [[ $service == "ovsdb-server" ]] && [[ "$systemctl_status" == "active(running)" ]]; then echo "PASS" >> $result_file else echo "FAIL" >> $result_file check_final_result exit 1 fi } test_systemctl_status_stopped() { service=$1 systemctl_status=$(systemctl status $service | grep Active | awk '{print $2$3}') if [[ $service == "openvswitch" ]] && [[ "$systemctl_status" == "inactive(dead)" ]]; then echo "PASS" >> $result_file else echo "FAIL" >> $result_file check_final_result exit 1 fi } test_process_crashed_restarted() { process=$1 vswitchd_pid1=$(pgrep vswitchd) ovsdb_pid1=$(pgrep vswitchd) if [[ $process == "vswitchd" ]]; then kill -SIGSEGV $vswitchd_pid1 elif [[ $process == "ovsdb" ]]; then kill -SIGSEGV $ovsdb_pid1 fi sleep 3 vswitchd_pid2=$(pgrep vswitchd) ovsdb_pid2=$(pgrep vswitchd) if [[ -z $vswitchd_pid2 ]] || [[ $vswitchd_pid2 == $vswitchd_pid1 ]]; then echo "FAIL" >> $result_file check_final_result exit 1 else echo "PASS" >> $result_file fi if [[ -z $ovsdb_pid2 ]] || [[ $ovsdb_pid2 == $ovsdb_pid1 ]]; then echo "FAIL" >> $result_file check_final_result exit 1 else echo "PASS" >> $result_file fi } test_process_started() { vswitchd_pid=$(pgrep vswitchd) ovsdb_pid=$(pgrep vswitchd) if [[ $vswitchd_pid ]]; then echo "PASS" >> $result_file else echo "FAIL" >> $result_file check_final_result exit 1 fi if [[ $ovsdb_pid ]]; then echo "PASS" >> $result_file else echo "FAIL" >> $result_file check_final_result exit 1 fi } test_process_stopped() { vswitchd_pid=$(pgrep vswitchd) ovsdb_pid=$(pgrep vswitchd) if [[ -z $vswitchd_pid ]]; then echo "PASS" >> $result_file else echo "FAIL" >> $result_file check_final_result exit 1 fi if [[ -z $ovsdb_pid ]]; then echo "PASS" >> $result_file else echo "FAIL" >> $result_file check_final_result exit 1 fi } get_process_pid() { process=$1 "$process"_pid=$(pgrep $process) } systemctl stop openvswitch sleep 3 ### Tests # start openvswitch systemctl start openvswitch sleep 3 test_systemctl_status_running openvswitch test_systemctl_status_running ovs-vswitchd test_systemctl_status_running ovsdb-server test_process_started # crash vswitchd count=1 while [ $count -le $iterations ]; do echo "Running vswitchd crash iteration $count of $iterations" test_process_crashed_restarted vswitchd let count++ done # crash ovsdb count=1 while [ $count -le $iterations ]; do echo "Running ovsdb crash iteration $count of $iterations" test_process_crashed_restarted ovsdb let count++ done check_final_result ######################################################### Tested several times against openvswitch-2.9.0-83.el7fdp.1.x86_64. Test never successfully proceeded beyond vswitch crash test. Below is an example test run: [root@netqe20 ~]# ./bz1684477.sh Running vswitchd crash iteration 1 of 50 Running vswitchd crash iteration 2 of 50 Running vswitchd crash iteration 3 of 50 Running vswitchd crash iteration 4 of 50 Running vswitchd crash iteration 5 of 50 Running vswitchd crash iteration 6 of 50 Running vswitchd crash iteration 7 of 50 Running vswitchd crash iteration 8 of 50 Running vswitchd crash iteration 9 of 50 Running vswitchd crash iteration 10 of 50 Running vswitchd crash iteration 11 of 50 Running vswitchd crash iteration 12 of 50 Running vswitchd crash iteration 13 of 50 13 crash iterations run of 50 total Overall Test Result: FAIL [root@netqe20 ~]# rpm -q openvswitch openvswitch-2.9.0-83.el7fdp.1.x86_64 Tested with openvswitch2.11-2.11.0-3.el7fdp.x86_64. All tests passed. Output below: [root@netqe21 ~]# ./bz1684477.sh Running vswitchd crash iteration 1 of 50 Running vswitchd crash iteration 2 of 50 Running vswitchd crash iteration 3 of 50 Running vswitchd crash iteration 4 of 50 Running vswitchd crash iteration 5 of 50 Running vswitchd crash iteration 6 of 50 Running vswitchd crash iteration 7 of 50 Running vswitchd crash iteration 8 of 50 Running vswitchd crash iteration 9 of 50 Running vswitchd crash iteration 10 of 50 Running vswitchd crash iteration 11 of 50 Running vswitchd crash iteration 12 of 50 Running vswitchd crash iteration 13 of 50 Running vswitchd crash iteration 14 of 50 Running vswitchd crash iteration 15 of 50 Running vswitchd crash iteration 16 of 50 Running vswitchd crash iteration 17 of 50 Running vswitchd crash iteration 18 of 50 Running vswitchd crash iteration 19 of 50 Running vswitchd crash iteration 20 of 50 Running vswitchd crash iteration 21 of 50 Running vswitchd crash iteration 22 of 50 Running vswitchd crash iteration 23 of 50 Running vswitchd crash iteration 24 of 50 Running vswitchd crash iteration 25 of 50 Running vswitchd crash iteration 26 of 50 Running vswitchd crash iteration 27 of 50 Running vswitchd crash iteration 28 of 50 Running vswitchd crash iteration 29 of 50 Running vswitchd crash iteration 30 of 50 Running vswitchd crash iteration 31 of 50 Running vswitchd crash iteration 32 of 50 Running vswitchd crash iteration 33 of 50 Running vswitchd crash iteration 34 of 50 Running vswitchd crash iteration 35 of 50 Running vswitchd crash iteration 36 of 50 Running vswitchd crash iteration 37 of 50 Running vswitchd crash iteration 38 of 50 Running vswitchd crash iteration 39 of 50 Running vswitchd crash iteration 40 of 50 Running vswitchd crash iteration 41 of 50 Running vswitchd crash iteration 42 of 50 Running vswitchd crash iteration 43 of 50 Running vswitchd crash iteration 44 of 50 Running vswitchd crash iteration 45 of 50 Running vswitchd crash iteration 46 of 50 Running vswitchd crash iteration 47 of 50 Running vswitchd crash iteration 48 of 50 Running vswitchd crash iteration 49 of 50 Running vswitchd crash iteration 50 of 50 Running ovsdb crash iteration 1 of 50 Running ovsdb crash iteration 2 of 50 Running ovsdb crash iteration 3 of 50 Running ovsdb crash iteration 4 of 50 Running ovsdb crash iteration 5 of 50 Running ovsdb crash iteration 6 of 50 Running ovsdb crash iteration 7 of 50 Running ovsdb crash iteration 8 of 50 Running ovsdb crash iteration 9 of 50 Running ovsdb crash iteration 10 of 50 Running ovsdb crash iteration 11 of 50 Running ovsdb crash iteration 12 of 50 Running ovsdb crash iteration 13 of 50 Running ovsdb crash iteration 14 of 50 Running ovsdb crash iteration 15 of 50 Running ovsdb crash iteration 16 of 50 Running ovsdb crash iteration 17 of 50 Running ovsdb crash iteration 18 of 50 Running ovsdb crash iteration 19 of 50 Running ovsdb crash iteration 20 of 50 Running ovsdb crash iteration 21 of 50 Running ovsdb crash iteration 22 of 50 Running ovsdb crash iteration 23 of 50 Running ovsdb crash iteration 24 of 50 Running ovsdb crash iteration 25 of 50 Running ovsdb crash iteration 26 of 50 Running ovsdb crash iteration 27 of 50 Running ovsdb crash iteration 28 of 50 Running ovsdb crash iteration 29 of 50 Running ovsdb crash iteration 30 of 50 Running ovsdb crash iteration 31 of 50 Running ovsdb crash iteration 32 of 50 Running ovsdb crash iteration 33 of 50 Running ovsdb crash iteration 34 of 50 Running ovsdb crash iteration 35 of 50 Running ovsdb crash iteration 36 of 50 Running ovsdb crash iteration 37 of 50 Running ovsdb crash iteration 38 of 50 Running ovsdb crash iteration 39 of 50 Running ovsdb crash iteration 40 of 50 Running ovsdb crash iteration 41 of 50 Running ovsdb crash iteration 42 of 50 Running ovsdb crash iteration 43 of 50 Running ovsdb crash iteration 44 of 50 Running ovsdb crash iteration 45 of 50 Running ovsdb crash iteration 46 of 50 Running ovsdb crash iteration 47 of 50 Running ovsdb crash iteration 48 of 50 Running ovsdb crash iteration 49 of 50 Running ovsdb crash iteration 50 of 50 50 crash iterations run of 50 total for each process Overall Test Result: PASS [root@netqe21 ~]# rpm -q openvswitch2.11 openvswitch2.11-2.11.0-3.el7fdp.x86_64 ######################################################### Marking as Verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0799 |