Bug 2062608

Summary: Failed to scaleup RHEL machine against OVN cluster due to jq tool is required by configure-ovs.sh
Product: OpenShift Container Platform Reporter: OpenShift BugZilla Robot <openshift-bugzilla-robot>
Component: NetworkingAssignee: Jaime Caamaño Ruiz <jcaamano>
Networking sub component: ovn-kubernetes QA Contact: Ross Brattain <rbrattai>
Status: CLOSED EOL Docs Contact:
Severity: urgent    
Priority: urgent CC: achernet, bzvonar, eglottma, jcaamano, mifiedle, openshift-bugzilla-robot, rbrattai, ross.b.brattain, rravaiol, vlaad, wking
Version: 4.10   
Target Milestone: ---   
Target Release: 4.7.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-21 19:18:06 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 2060476, 2062655    
Bug Blocks:    

Comment 7 Ross Brattain 2022-10-25 20:10:26 UTC 
jq is used with DHCP only.

Failed because RHEL 7.9 iproute2 package is too old for `-j` JSON output.

sh-4.2# rpm -qf `which jq`
jq-1.6-2.el7.x86_64
sh-4.2# rpm -qf `which ip`
iproute-4.11.0-30.el7.x86_64
sh-4.2# ip -j a show
Option "-j" is unknown, try "ip -help".

Oct 25 19:31:44.196914 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] + nmcli connection show ovs-if-br-ex
Oct 25 19:31:44.213491 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] + ovs-vsctl --timeout=30 --if-exists destroy interface br-ex
Oct 25 19:31:44.215692 ip-10-0-50-19.compute.internal ovs-vsctl[1696] ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=30 --if-exists destroy interface br-ex
Oct 25 19:31:44.219696 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] + nmcli --fields ipv4.method,ipv6.method conn show 5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03
Oct 25 19:31:44.220813 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] + grep manual
Oct 25 19:31:44.239952 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] + extra_if_brex_args=
Oct 25 19:31:44.241115 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] ++ ip -j a show dev eth0
Oct 25 19:31:44.242234 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] ++ jq '.[0].addr_info | map(. | select(.family == "inet")) | length'
Oct 25 19:31:44.243309 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] Option "-j" is unknown, try "ip -help".
Oct 25 19:31:44.299180 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] + num_ipv4_addrs=
Oct 25 19:31:44.300304 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] + '[' '' -gt 0 ']'
Oct 25 19:31:44.301369 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] /usr/local/bin/configure-ovs.sh: line 290: [: : integer expression expected
Oct 25 19:31:44.302421 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] ++ ip -j a show dev eth0
Oct 25 19:31:44.304299 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] ++ jq '.[0].addr_info | map(. | select(.family == "inet6" and .scope != "link")) | length'
Oct 25 19:31:44.305368 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] Option "-j" is unknown, try "ip -help".
Oct 25 19:31:44.344459 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] + num_ip6_addrs=
Oct 25 19:31:44.345573 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] + '[' '' -gt 0 ']'
Oct 25 19:31:44.346627 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] /usr/local/bin/configure-ovs.sh: line 297: [: : integer expression expected
Oct 25 19:31:44.347681 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] ++ nmcli --get-values ipv4.dhcp-client-id conn show 5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03
Oct 25 19:31:44.364802 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] + dhcp_client_id=
Oct 25 19:31:44.365886 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] + '[' -n '' ']'
Oct 25 19:31:44.366944 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] ++ nmcli --get-values ipv6.dhcp-duid conn show 5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03
Oct 25 19:31:44.384074 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] + dhcp6_client_id=
Oct 25 19:31:44.385138 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] + '[' -n '' ']'
Oct 25 19:31:44.386165 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] ++ nmcli --get-values ipv6.addr-gen-mode conn show 5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03
Oct 25 19:31:44.404292 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] + ipv6_addr_gen_mode=stable-privacy
Oct 25 19:31:44.405964 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] + '[' -n stable-privacy ']'
Oct 25 19:31:44.407620 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] + extra_if_brex_args+='ipv6.addr-gen-mode stable-privacy '
Oct 25 19:31:44.411299 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] + add_nm_conn type ovs-interface slave-type ovs-port conn.interface br-ex master c43366b8-a71f-4d7f-8397-f3d951a912f1 con-name ovs-if-br-ex 802-3-ethernet.mtu 9001 802-3-ethernet.clon>
Oct 25 19:31:44.413957 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] + nmcli c add type ovs-interface slave-type ovs-port conn.interface br-ex master c43366b8-a71f-4d7f-8397-f3d951a912f1 con-name ovs-if-br-ex 802-3-ethernet.mtu 9001 802-3-ethernet.clon>
Oct 25 19:31:44.430494 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] Connection 'ovs-if-br-ex' (112e9400-3ab4-4c2f-a87b-da53c620620e) successfully added.
Oct 25 19:31:44.431866 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] + configure_driver_options eth0
Oct 25 19:31:44.432938 ip-10-0-50-19.compute.internal configure-ovs.sh[1256] + intf=eth0


sh-4.2# which jq
/usr/bin/jq
sh-4.2# rpm -qf `which jq`
jq-1.6-2.el7.x86_64

PRETTY_NAME="Red Hat Enterprise Linux Server 7.9 (Maipo)"

The node comes online anyway, though. 


ip-10-0-50-19.compute.internal    Ready    worker   35m   v1.20.15+98b2293   10.0.50.19    <none>        Red Hat Enterprise Linux Server 7.9 (Maipo)                    3.10.0-1160.76.1.el7.x86_64    cri-o://1.20.9-4.rhaos4.7.gitfa57563.el7
Comment 8 Ross Brattain 2022-10-25 20:11:39 UTC 
^ on 4.7.0-0.nightly-2022-10-17-061132
Comment 12 Ross Brattain 2022-11-03 02:31:47 UTC 
only affects DHCP.

Tested on 4.7.0-0.ci.test-2022-11-02-164550-ci-ln-m9b4qdt-latest


vSphere UPI RHEL 7.9 OVN mode=active-backup,fail_over_mac=0,miimon=100

ovs-config-172.31.249.32.log-Nov 02 15:36:41 o47vb33-9v6g4-rhel-0 configure-ovs.sh[1512]: + extra_if_brex_args=
ovs-config-172.31.249.32.log-Nov 02 15:36:41 o47vb33-9v6g4-rhel-0 configure-ovs.sh[1512]: ++ ip a show dev bond0
ovs-config-172.31.249.32.log-Nov 02 15:36:41 o47vb33-9v6g4-rhel-0 configure-ovs.sh[1512]: ++ grep -E '^[[:blank:]]*inet\b'
ovs-config-172.31.249.32.log-Nov 02 15:36:41 o47vb33-9v6g4-rhel-0 configure-ovs.sh[1512]: ++ wc -l
ovs-config-172.31.249.32.log:Nov 02 15:36:41 o47vb33-9v6g4-rhel-0 configure-ovs.sh[1512]: + num_ipv4_addrs=1
ovs-config-172.31.249.32.log-Nov 02 15:36:41 o47vb33-9v6g4-rhel-0 configure-ovs.sh[1512]: + '[' 1 -gt 0 ']'
ovs-config-172.31.249.32.log-Nov 02 15:36:41 o47vb33-9v6g4-rhel-0 configure-ovs.sh[1512]: + extra_if_brex_args+='ipv4.may-fail no '
ovs-config-172.31.249.32.log-Nov 02 15:36:41 o47vb33-9v6g4-rhel-0 configure-ovs.sh[1512]: ++ ip a show dev bond0
ovs-config-172.31.249.32.log-Nov 02 15:36:41 o47vb33-9v6g4-rhel-0 configure-ovs.sh[1512]: ++ grep -E '^[[:blank:]]*inet6\b'
ovs-config-172.31.249.32.log-Nov 02 15:36:41 o47vb33-9v6g4-rhel-0 configure-ovs.sh[1512]: ++ grep -v '\bscope link\b'
ovs-config-172.31.249.32.log-Nov 02 15:36:41 o47vb33-9v6g4-rhel-0 configure-ovs.sh[1512]: ++ wc -l
ovs-config-172.31.249.32.log:Nov 02 15:36:41 o47vb33-9v6g4-rhel-0 configure-ovs.sh[1512]: + num_ip6_addrs=0
ovs-config-172.31.249.32.log-Nov 02 15:36:41 o47vb33-9v6g4-rhel-0 configure-ovs.sh[1512]: + '[' 0 -gt 0 ']'
ovs-config-172.31.249.32.log-Nov 02 15:36:41 o47vb33-9v6g4-rhel-0 configure-ovs.sh[1512]: ++ nmcli --get-values ipv4.dhcp-client-id conn show ba5b0e8a-9c72-3fe5-b1bd-cfbeabe5c45c
ovs-config-172.31.249.32.log-Nov 02 15:36:41 o47vb33-9v6g4-rhel-0 configure-ovs.sh[1512]: + dhcp_client_id=
ovs-config-172.31.249.32.log-Nov 02 15:36:41 o47vb33-9v6g4-rhel-0 configure-ovs.sh[1512]: + '[' -n '' ']'


libvirt IPI RHCOS DHCP mode=802.3ad,fail_over_mac=0,miimon=100

ovs-config-192.168.123.58.log-Nov 03 01:12:03 master-0-0.o47r1db-0.qe.lab.redhat.com configure-ovs.sh[2120]: + extra_if_brex_args=
ovs-config-192.168.123.58.log-Nov 03 01:12:03 master-0-0.o47r1db-0.qe.lab.redhat.com configure-ovs.sh[2120]: ++ ip a show dev bond0
ovs-config-192.168.123.58.log-Nov 03 01:12:03 master-0-0.o47r1db-0.qe.lab.redhat.com configure-ovs.sh[2120]: ++ grep -E '^[[:blank:]]*inet\b'
ovs-config-192.168.123.58.log-Nov 03 01:12:03 master-0-0.o47r1db-0.qe.lab.redhat.com configure-ovs.sh[2120]: ++ wc -l
ovs-config-192.168.123.58.log:Nov 03 01:12:03 master-0-0.o47r1db-0.qe.lab.redhat.com configure-ovs.sh[2120]: + num_ipv4_addrs=1
ovs-config-192.168.123.58.log-Nov 03 01:12:03 master-0-0.o47r1db-0.qe.lab.redhat.com configure-ovs.sh[2120]: + '[' 1 -gt 0 ']'
ovs-config-192.168.123.58.log-Nov 03 01:12:03 master-0-0.o47r1db-0.qe.lab.redhat.com configure-ovs.sh[2120]: + extra_if_brex_args+='ipv4.may-fail no '
ovs-config-192.168.123.58.log-Nov 03 01:12:03 master-0-0.o47r1db-0.qe.lab.redhat.com configure-ovs.sh[2120]: ++ ip a show dev bond0
ovs-config-192.168.123.58.log-Nov 03 01:12:03 master-0-0.o47r1db-0.qe.lab.redhat.com configure-ovs.sh[2120]: ++ grep -E '^[[:blank:]]*inet6\b'
ovs-config-192.168.123.58.log-Nov 03 01:12:03 master-0-0.o47r1db-0.qe.lab.redhat.com configure-ovs.sh[2120]: ++ grep -v '\bscope link\b'
ovs-config-192.168.123.58.log-Nov 03 01:12:03 master-0-0.o47r1db-0.qe.lab.redhat.com configure-ovs.sh[2120]: ++ wc -l
ovs-config-192.168.123.58.log:Nov 03 01:12:03 master-0-0.o47r1db-0.qe.lab.redhat.com configure-ovs.sh[2120]: + num_ip6_addrs=1
ovs-config-192.168.123.58.log-Nov 03 01:12:03 master-0-0.o47r1db-0.qe.lab.redhat.com configure-ovs.sh[2120]: + '[' 1 -gt 0 ']'
ovs-config-192.168.123.58.log-Nov 03 01:12:03 master-0-0.o47r1db-0.qe.lab.redhat.com configure-ovs.sh[2120]: + extra_if_brex_args+='ipv6.may-fail no '
ovs-config-192.168.123.58.log-Nov 03 01:12:03 master-0-0.o47r1db-0.qe.lab.redhat.com configure-ovs.sh[2120]: ++ nmcli --get-values ipv4.dhcp-client-id conn show a7cc816d-3dbd-34c5-9902-d6b2f2956d92
ovs-config-192.168.123.58.log-Nov 03 01:12:03 master-0-0.o47r1db-0.qe.lab.redhat.com configure-ovs.sh[2120]: + dhcp_client_id=
Comment 17 Vikas Laad 2022-11-21 19:18:06 UTC 
OCP 4.7 is EOL now https://access.redhat.com/errata/RHBA-2022:7334

Closing all the open bugs.