Bug 2182453
| Summary: | ovn-kubernetes br-ex ovs-interface not sending DHCP on activation | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Quique Llorente <ellorent> |
| Component: | NetworkManager | Assignee: | NetworkManager Development Team <nm-team> |
| Status: | CLOSED NOTABUG | QA Contact: | Desktop QE <desktop-qa-list> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 9.2 | CC: | bgalvani, lrintel, phoracek, rkhan, sfaye, sukulkar, till |
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-04-18 07:29:49 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
The packet capture looks weird. First, while the log indicates that NM is sending multiple DHCP DISCOVERs, the capture only has a REQUEST followed by an ACK and then many REQUESTs again. Also, all the REQUESTs appear very close to each other and they look identical. One possible explanation could be that there is a loop in the network topology which creates a packet storm. Can you please investigate this, and perhaps also do a packet capture on enp1s0 on the client machine to see what NM is sending? (In reply to Beniamino Galvani from comment #4) > The packet capture looks weird. First, while the log indicates that NM is > sending multiple DHCP DISCOVERs, the capture only has a REQUEST followed by > an ACK and then many REQUESTs again. Also, all the REQUESTs appear very > close to each other and they look identical. One possible explanation could > be that there is a loop in the network topology which creates a packet storm. > > Can you please investigate this, and perhaps also do a packet capture on > enp1s0 on the client machine to see what NM is sending? Those DHCP messages are from the beggining and end of the ovs-configuration script, the flow is like this 1 - enp1s0 is configured 2 - DHCP DISCOVER/REQUEST is received 3 - br-ex ovs-interface is created cloning the enp1s0 mac address 4 - nmcli con up br-ex is retried 10 times (NetworkManager waits 45 seconds each) 5 - DHCP is not received 6 - ovs-configuration do a "rollback" and remove the ovs artifacts and set enp1s0 up 7 - DHCP DISCOVER/REQUEST is received What you see there is the DHCP from 1 and 5 We are preparing and env to investigate @bgalvani we have manage to capture from the node primary interface although is different deployment so mac addresses and ip will not match with journal attached.
We see the expected behaviour
This is at first boot before br-ex take over
dropped privs to tcpdump
tcpdump: listening on tap0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
18:56:29.390646 0a:58:0a:83:00:6b > Broadcast, ethertype IPv4 (0x0800), length 324: (tos 0xc0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 310)
0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 0a:58:0a:83:00:6b, length 282, xid 0x17e82d7f, secs 1, Flags [none] (0x0000)
Client-Ethernet-Address 0a:58:0a:83:00:6b
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Request
Client-ID (61), length 7: ether 0a:58:0a:83:00:6b
Parameter-Request (55), length 17:
Subnet-Mask (1), Time-Zone (2), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), MTU (26), BR (28), Classless-Static-Route (121)
Default-Gateway (3), Static-Route (33), YD (40), YS (41)
NTP (42), Unknown (119), Classless-Static-Route-Microsoft (249), Unknown (252)
RP (17)
MSZ (57), length 2: 576
Requested-IP (50), length 4: 10.131.0.107
18:56:29.390809 7e:27:81:b2:f0:01 > Broadcast, ethertype IPv4 (0x0800), length 516: (tos 0x0, ttl 64, id 41420, offset 0, flags [DF], proto UDP (17), length 502)
169.254.75.10.bootps > 255.255.255.255.bootpc: [udp sum ok] BOOTP/DHCP, Reply, length 474, xid 0x17e82d7f, Flags [none] (0x0000)
Your-IP 10.131.0.107
Client-Ethernet-Address 0a:58:0a:83:00:6b
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: ACK
Server-ID (54), length 4: 169.254.75.10
Lease-Time (51), length 4: 86313600
Subnet-Mask (1), length 4: 255.255.254.0
Default-Gateway (3), length 4: 10.131.0.1
Classless-Static-Route (121), length 5: (default:10.131.0.1)
Unknown (119), length 114: 8293,12901,11619,27765,29556,25970,29485,31331,25194,27949,25976,24941,28780,25901,13618,26216,30211,29558,25351,25452,30067,29797,29189,27759,25441,27648,883,30307,1891,27765,29556,25970,1388,28515,24940,7,25452,30067,29797,29189,27759,25441,27648,2421,29485,30565,29556,11570,1891,28525,28789,29797,2153,28276,25970,28257,27648
Hostname (12), length 19: "example-52fhv-lnq4w"
Domain-Name (15), length 50: "e2e-clusters-zcbjm-example-52fhv.svc.cluster.local"
Domain-Name-Server (6), length 4: 172.30.0.10
MTU (26), length 2: 8901
Then we have "nmcli con up ovs-if-br-ex" trying to take over it but there is no DHCP request at all
<EMPTY>
After that is the "rollback process" where the primary interace is removed from ovs-bridge and re-configured then we see expected DHCP
19:04:53.218263 0a:58:0a:83:00:6b > Broadcast, ethertype IPv4 (0x0800), length 324: (tos 0xc0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 310)
0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 0a:58:0a:83:00:6b, length 282, xid 0xe119a459, secs 1, Flags [none] (0x0000)
Client-Ethernet-Address 0a:58:0a:83:00:6b
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Request
Client-ID (61), length 7: ether 0a:58:0a:83:00:6b
Parameter-Request (55), length 17:
Subnet-Mask (1), Time-Zone (2), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), MTU (26), BR (28), Classless-Static-Route (121)
Default-Gateway (3), Static-Route (33), YD (40), YS (41)
NTP (42), Unknown (119), Classless-Static-Route-Microsoft (249), Unknown (252)
RP (17)
MSZ (57), length 2: 576
Requested-IP (50), length 4: 10.131.0.107
19:04:53.218447 7e:27:81:b2:f0:01 > Broadcast, ethertype IPv4 (0x0800), length 516: (tos 0x0, ttl 64, id 54538, offset 0, flags [DF], proto UDP (17), length 502)
169.254.75.10.bootps > 255.255.255.255.bootpc: [udp sum ok] BOOTP/DHCP, Reply, length 474, xid 0xe119a459, Flags [none] (0x0000)
Your-IP 10.131.0.107
Client-Ethernet-Address 0a:58:0a:83:00:6b
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: ACK
Server-ID (54), length 4: 169.254.75.10
Lease-Time (51), length 4: 86313600
Domain-Name-Server (6), length 4: 172.30.0.10
MTU (26), length 2: 8901
Subnet-Mask (1), length 4: 255.255.254.0
Default-Gateway (3), length 4: 10.131.0.1
Classless-Static-Route (121), length 5: (default:10.131.0.1)
Unknown (119), length 114: 8293,12901,11619,27765,29556,25970,29485,31331,25194,27949,25976,24941,28780,25901,13618,26216,30211,29558,25351,25452,30067,29797,29189,27759,25441,27648,883,30307,1891,27765,29556,25970,1388,28515,24940,7,25452,30067,29797,29189,27759,25441,27648,2421,29485,30565,29556,11570,1891,28525,28789,29797,2153,28276,25970,28257,27648
Hostname (12), length 19: "example-52fhv-lnq4w"
Domain-Name (15), length 50: "e2e-clusters-zcbjm-example-52fhv.svc.cluster.local"
Also I have see that we have a NM dispatcher scritp related ovs I don't know if it can be related /etc/NetworkManager/dispatcher.d/pre-up.d/10-ofport-request.sh #!/bin/bash # Set interface ofport_request to guarantee stable ofport numbers. This is important for flow matches. # Otherwise, another ofport number is assigned to the interface on every restart of NetworkManager. # This script will build an associative array INTERFACE_NAME->ofport_request and will save it to file CONFIGURATION_FILE. # When an interface is brought up, this will reuse the value from the associative array if such a value exists. # Otherwise, this will try to use the current ofport value. If the ofport value is already reserved, then # this uses the lowest available numerical value, instead. This is the full dispatchers list /etc/NetworkManager/dispatcher.d/ /etc/NetworkManager/dispatcher.d/20-coreos-chrony-dhcp /etc/NetworkManager/dispatcher.d/90-console-login-helper-messages-gensnippet_if /etc/NetworkManager/dispatcher.d/no-wait.d /etc/NetworkManager/dispatcher.d/pre-down.d /etc/NetworkManager/dispatcher.d/pre-up.d /etc/NetworkManager/dispatcher.d/pre-up.d/10-ofport-request.sh /etc/NetworkManager/dispatcher.d/99-vsphere-disable-tx-udp-tnl > Then we have "nmcli con up ovs-if-br-ex" trying to take over it but there is no DHCP request at all > <EMPTY> Ok, this means that the requests are not forwarded from br-ex to the physical interface. From the log attached it seems that all interfaces are configured properly, but to double-check, is it possible for you to show the output of these commands when DHCP is running? ip addr ovs-vsctl show Also, please try to remove the `10-ofport-request.sh` dispatcher script to see if that makes any difference. To add some context ovs-configuration.service [Unit] Description=Configures OVS with proper host networking configuration # Removal of this file signals firstboot completion ConditionPathExists=!/etc/ignition-machine-config-encapsulated.json # This service is used to move a physical NIC into OVS and reconfigure OVS to use the host IP Requires=openvswitch.service Wants=NetworkManager-wait-online.service After=NetworkManager-wait-online.service openvswitch.service network.service nodeip-configuration.service Before=network-online.target kubelet.service crio.service node-valid-hostname.service [Service] # Need oneshot to delay kubelet Type=oneshot ExecStart=/usr/local/bin/configure-ovs.sh OVNKubernetes StandardOutput=journal+console StandardError=journal+console [Install] WantedBy=network-online.target NetworkManager dispatcher service [Unit] Description=Network Manager Script Dispatcher Service [Service] Type=dbus BusName=org.freedesktop.nm_dispatcher ExecStart=/usr/libexec/nm-dispatcher NotifyAccess=main # Enable debug logging in dispatcher service. Note that dispatcher # also honors debug logging requests from NetworkManager, so you # can also control logging requests with # `nmcli general logging domain DISPATCHER level TRACE`. #Environment=NM_DISPATCHER_DEBUG_LOG=1 # We want to allow scripts to spawn long-running daemons, so tell # systemd to not clean up when nm-dispatcher exits KillMode=process [Install] Alias=dbus-org.freedesktop.nm-dispatcher.service We see that at a working env the dispatcher has started before ovs-configuration Also looks like there is a race between networkmanager service unit readiness and dbus endpoint https://bugzilla.redhat.com/show_bug.cgi?id=2177188 This is the ovs-vsctl show and ip addr between nmcli con up retries, I don't see br-ex there
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[1204]: + for i in {1..10}
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[1204]: + ip addr
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2138]: 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2138]: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2138]: inet 127.0.0.1/8 scope host lo
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2138]: valid_lft forever preferred_lft forever
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2138]: inet6 ::1/128 scope host
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2138]: valid_lft forever preferred_lft forever
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2138]: 2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8901 qdisc fq_codel master ovs-system state UP group default qlen 1000
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2138]: link/ether 0a:58:0a:83:00:6c brd ff:ff:ff:ff:ff:ff
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2138]: 3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2138]: link/ether c6:46:2f:0d:81:6c brd ff:ff:ff:ff:ff:ff
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2138]: 4: ovn-k8s-mp0: <BROADCAST,MULTICAST> mtu 8801 qdisc noop state DOWN group default qlen 1000
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2138]: link/ether 62:85:1c:e0:f1:6d brd ff:ff:ff:ff:ff:ff
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2138]: 5: br-int: <BROADCAST,MULTICAST> mtu 8801 qdisc noop state DOWN group default qlen 1000
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2138]: link/ether 16:eb:70:f0:3d:ae brd ff:ff:ff:ff:ff:ff
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[1204]: + ovs-vsctl show
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2139]: 9543d237-84ff-43a0-ba00-959bc51d02d0
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2139]: Bridge br-int
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2139]: fail_mode: secure
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2139]: datapath_type: system
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2139]: Port ovn-k8s-mp0
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2139]: Interface ovn-k8s-mp0
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2139]: type: internal
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2139]: Port br-int
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2139]: Interface br-int
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2139]: type: internal
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2139]: Bridge br-ex
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2139]: Port enp1s0
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2139]: Interface enp1s0
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2139]: type: system
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[2139]: ovs_version: "3.1.1"
Mar 30 14:28:23 localhost.localdomain configure-ovs.sh[1204]: + echo 'Attempt 10 to bring up connection ovs-if-br-ex'
We manage to capture tcpdump inside the node at ovs-configuration
Looks like br-ex is sending the DHCP request but it's not traversiong the ovs-bridge and it's not re-transmited to enp1s0
15:12:26.787915 br-ex Out ifindex 6 0a:58:0a:83:00:6c ethertype IPv4 (0x0800), length 324: (tos 0xc0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 304)
0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 0a:58:0a:83:00:6c, length 276, xid 0xc1f40370, secs 1, Flags [none] (0x0000)
Client-Ethernet-Address 0a:58:0a:83:00:6c
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Client-ID (61), length 7: ether 0a:58:0a:83:00:6c
Parameter-Request (55), length 17:
Subnet-Mask (1), Time-Zone (2), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), MTU (26), BR (28), Classless-Static-Route (121)
Default-Gateway (3), Static-Route (33), YD (40), YS (41)
NTP (42), Unknown (119), Classless-Static-Route-Microsoft (249), Unknown (252)
RP (17)
MSZ (57), length 2: 576
15:12:29.222415 br-ex Out ifindex 6 0a:58:0a:83:00:6c ethertype IPv4 (0x0800), length 324: (tos 0xc0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 304)
0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 0a:58:0a:83:00:6c, length 276, xid 0xe9e4852b, secs 2, Flags [none] (0x0000)
Client-Ethernet-Address 0a:58:0a:83:00:6c
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Client-ID (61), length 7: ether 0a:58:0a:83:00:6c
Parameter-Request (55), length 17:
Subnet-Mask (1), Time-Zone (2), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), MTU (26), BR (28), Classless-Static-Route (121)
Default-Gateway (3), Static-Route (33), YD (40), YS (41)
NTP (42), Unknown (119), Classless-Static-Route-Microsoft (249), Unknown (252)
RP (17)
MSZ (57), length 2: 576
15:12:33.269742 br-ex Out ifindex 6 0a:58:0a:83:00:6c ethertype IPv4 (0x0800), length 324: (tos 0xc0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 304)
0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 0a:58:0a:83:00:6c, length 276, xid 0x78fab960, secs 6, Flags [none] (0x0000)
Client-Ethernet-Address 0a:58:0a:83:00:6c
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Client-ID (61), length 7: ether 0a:58:0a:83:00:6c
Parameter-Request (55), length 17:
Subnet-Mask (1), Time-Zone (2), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), MTU (26), BR (28), Classless-Static-Route (121)
Default-Gateway (3), Static-Route (33), YD (40), YS (41)
NTP (42), Unknown (119), Classless-Static-Route-Microsoft (249), Unknown (252)
RP (17)
MSZ (57), length 2: 576
15:12:42.247225 br-ex Out ifindex 6 0a:58:0a:83:00:6c ethertype IPv4 (0x0800), length 324: (tos 0xc0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 304)
0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 0a:58:0a:83:00:6c, length 276, xid 0xac365562, secs 15, Flags [none] (0x0000)
Client-Ethernet-Address 0a:58:0a:83:00:6c
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Client-ID (61), length 7: ether 0a:58:0a:83:00:6c
Parameter-Request (55), length 17:
Subnet-Mask (1), Time-Zone (2), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), MTU (26), BR (28), Classless-Static-Route (121)
Default-Gateway (3), Static-Route (33), YD (40), YS (41)
NTP (42), Unknown (119), Classless-Static-Route-Microsoft (249), Unknown (252)
RP (17)
MSZ (57), length 2: 576
15:12:58.686920 br-ex Out ifindex 6 0a:58:0a:83:00:6c ethertype IPv4 (0x0800), length 324: (tos 0xc0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 304)
0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 0a:58:0a:83:00:6c, length 276, xid 0x6e399967, secs 31, Flags [none] (0x0000)
Client-Ethernet-Address 0a:58:0a:83:00:6c
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Client-ID (61), length 7: ether 0a:58:0a:83:00:6c
Parameter-Request (55), length 17:
Subnet-Mask (1), Time-Zone (2), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), MTU (26), BR (28), Classless-Static-Route (121)
Default-Gateway (3), Static-Route (33), YD (40), YS (41)
NTP (42), Unknown (119), Classless-Static-Route-Microsoft (249), Unknown (252)
RP (17)
MSZ (57), length 2: 576
15:13:16.824842 br-ex Out ifindex 7 0a:58:0a:83:00:6c ethertype IPv4 (0x0800), length 324: (tos 0xc0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 304)
0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 0a:58:0a:83:00:6c, length 276, xid 0x617ad2ae, secs 1, Flags [none] (0x0000)
Client-Ethernet-Address 0a:58:0a:83:00:6c
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Client-ID (61), length 7: ether 0a:58:0a:83:00:6c
Parameter-Request (55), length 17:
Subnet-Mask (1), Time-Zone (2), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), MTU (26), BR (28), Classless-Static-Route (121)
Default-Gateway (3), Static-Route (33), YD (40), YS (41)
NTP (42), Unknown (119), Classless-Static-Route-Microsoft (249), Unknown (252)
RP (17)
MSZ (57), length 2: 576
15:13:18.828109 br-ex Out ifindex 7 0a:58:0a:83:00:6c ethertype IPv4 (0x0800), length 324: (tos 0xc0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 304)
0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 0a:58:0a:83:00:6c, length 276, xid 0xd96f4689, secs 2, Flags [none] (0x0000)
Client-Ethernet-Address 0a:58:0a:83:00:6c
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Client-ID (61), length 7: ether 0a:58:0a:83:00:6c
Parameter-Request (55), length 17:
Subnet-Mask (1), Time-Zone (2), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), MTU (26), BR (28), Classless-Static-Route (121)
Default-Gateway (3), Static-Route (33), YD (40), YS (41)
NTP (42), Unknown (119), Classless-Static-Route-Microsoft (249), Unknown (252)
RP (17)
MSZ (57), length 2: 576
15:13:22.928433 br-ex Out ifindex 7 0a:58:0a:83:00:6c ethertype IPv4 (0x0800), length 324: (tos 0xc0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 304)
0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 0a:58:0a:83:00:6c, length 276, xid 0xb20ea058, secs 6, Flags [none] (0x0000)
Client-Ethernet-Address 0a:58:0a:83:00:6c
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Client-ID (61), length 7: ether 0a:58:0a:83:00:6c
Parameter-Request (55), length 17:
Subnet-Mask (1), Time-Zone (2), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), MTU (26), BR (28), Classless-Static-Route (121)
Default-Gateway (3), Static-Route (33), YD (40), YS (41)
NTP (42), Unknown (119), Classless-Static-Route-Microsoft (249), Unknown (252)
RP (17)
MSZ (57), length 2: 576
15:13:31.725297 br-ex Out ifindex 7 0a:58:0a:83:00:6c ethertype IPv4 (0x0800), length 324: (tos 0xc0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 304)
0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 0a:58:0a:83:00:6c, length 276, xid 0xe6fd42c0, secs 14, Flags [none] (0x0000)
Client-Ethernet-Address 0a:58:0a:83:00:6c
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Client-ID (61), length 7: ether 0a:58:0a:83:00:6c
Parameter-Request (55), length 17:
Subnet-Mask (1), Time-Zone (2), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), MTU (26), BR (28), Classless-Static-Route (121)
Default-Gateway (3), Static-Route (33), YD (40), YS (41)
NTP (42), Unknown (119), Classless-Static-Route-Microsoft (249), Unknown (252)
RP (17)
MSZ (57), length 2: 576
15:13:47.966048 br-ex Out ifindex 7 0a:58:0a:83:00:6c ethertype IPv4 (0x0800), length 324: (tos 0xc0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 304)
0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 0a:58:0a:83:00:6c, length 276, xid 0x62dbfa1b, secs 31, Flags [none] (0x0000)
Client-Ethernet-Address 0a:58:0a:83:00:6c
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Client-ID (61), length 7: ether 0a:58:0a:83:00:6c
Parameter-Request (55), length 17:
Subnet-Mask (1), Time-Zone (2), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), MTU (26), BR (28), Classless-Static-Route (121)
Default-Gateway (3), Static-Route (33), YD (40), YS (41)
NTP (42), Unknown (119), Classless-Static-Route-Microsoft (249), Unknown (252)
RP (17)
MSZ (57), length 2: 576
15:14:06.817615 br-ex Out ifindex 8 0a:58:0a:83:00:6c ethertype IPv4 (0x0800), length 324: (tos 0xc0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 304)
0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 0a:58:0a:83:00:6c, length 276, xid 0xaf1fcf75, secs 1, Flags [none] (0x0000)
Client-Ethernet-Address 0a:58:0a:83:00:6c
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Client-ID (61), length 7: ether 0a:58:0a:83:00:6c
Parameter-Request (55), length 17:
Subnet-Mask (1), Time-Zone (2), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), MTU (26), BR (28), Classless-Static-Route (121)
Default-Gateway (3), Static-Route (33), YD (40), YS (41)
NTP (42), Unknown (119), Classless-Static-Route-Microsoft (249), Unknown (252)
RP (17)
MSZ (57), length 2: 576
15:14:08.931777 br-ex Out ifindex 8 0a:58:0a:83:00:6c ethertype IPv4 (0x0800), length 324: (tos 0xc0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 304)
0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 0a:58:0a:83:00:6c, length 276, xid 0xe6fcc5b7, secs 2, Flags [none] (0x0000)
Client-Ethernet-Address 0a:58:0a:83:00:6c
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Client-ID (61), length 7: ether 0a:58:0a:83:00:6c
Parameter-Request (55), length 17:
Subnet-Mask (1), Time-Zone (2), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), MTU (26), BR (28), Classless-Static-Route (121)
Default-Gateway (3), Static-Route (33), YD (40), YS (41)
NTP (42), Unknown (119), Classless-Static-Route-Microsoft (249), Unknown (252)
RP (17)
MSZ (57), length 2: 576
15:14:13.147699 br-ex Out ifindex 8 0a:58:0a:83:00:6c ethertype IPv4 (0x0800), length 324: (tos 0xc0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 304)
0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 0a:58:0a:83:00:6c, length 276, xid 0x20fe7193, secs 6, Flags [none] (0x0000)
Client-Ethernet-Address 0a:58:0a:83:00:6c
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Client-ID (61), length 7: ether 0a:58:0a:83:00:6c
Parameter-Request (55), length 17:
Subnet-Mask (1), Time-Zone (2), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), MTU (26), BR (28), Classless-Static-Route (121)
Default-Gateway (3), Static-Route (33), YD (40), YS (41)
NTP (42), Unknown (119), Classless-Static-Route-Microsoft (249), Unknown (252)
RP (17)
MSZ (57), length 2: 576
15:14:21.199604 br-ex Out ifindex 8 0a:58:0a:83:00:6c ethertype IPv4 (0x0800), length 324: (tos 0xc0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 304)
0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 0a:58:0a:83:00:6c, length 276, xid 0x40dfe96d, secs 14, Flags [none] (0x0000)
Client-Ethernet-Address 0a:58:0a:83:00:6c
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Client-ID (61), length 7: ether 0a:58:0a:83:00:6c
Parameter-Request (55), length 17:
Subnet-Mask (1), Time-Zone (2), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), MTU (26), BR (28), Classless-Static-Route (121)
Default-Gateway (3), Static-Route (33), YD (40), YS (41)
NTP (42), Unknown (119), Classless-Static-Route-Microsoft (249), Unknown (252)
RP (17)
MSZ (57), length 2: 576
15:14:37.227004 br-ex Out ifindex 8 0a:58:0a:83:00:6c ethertype IPv4 (0x0800), length 324: (tos 0xc0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 304)
0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 0a:58:0a:83:00:6c, length 276, xid 0x31c782da, secs 30, Flags [none] (0x0000)
Client-Ethernet-Address 0a:58:0a:83:00:6c
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Client-ID (61), length 7: ether 0a:58:0a:83:00:6c
Parameter-Request (55), length 17:
Subnet-Mask (1), Time-Zone (2), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), MTU (26), BR (28), Classless-Static-Route (121)
Default-Gateway (3), Static-Route (33), YD (40), YS (41)
NTP (42), Unknown (119), Classless-Static-Route-Microsoft (249), Unknown (252)
RP (17)
MSZ (57), length 2: 576
15:14:57.036422 enp1s0 Out ifindex 2 0a:58:0a:83:00:6c ethertype IPv4 (0x0800), length 330: (tos 0xc0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 310)
0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 0a:58:0a:83:00:6c, length 282, xid 0x61b2cbb2, secs 1, Flags [none] (0x0000)
Client-Ethernet-Address 0a:58:0a:83:00:6c
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Request
Client-ID (61), length 7: ether 0a:58:0a:83:00:6c
Parameter-Request (55), length 17:
Subnet-Mask (1), Time-Zone (2), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), MTU (26), BR (28), Classless-Static-Route (121)
Default-Gateway (3), Static-Route (33), YD (40), YS (41)
NTP (42), Unknown (119), Classless-Static-Route-Microsoft (249), Unknown (252)
RP (17)
MSZ (57), length 2: 576
Requested-IP (50), length 4: 10.131.0.108
15:14:57.036928 enp1s0 B ifindex 2 2e:95:3c:40:be:d0 ethertype IPv4 (0x0800), length 522: (tos 0x0, ttl 64, id 39481, offset 0, flags [DF], proto UDP (17), length 502)
169.254.75.10.bootps > 255.255.255.255.bootpc: [udp sum ok] BOOTP/DHCP, Reply, length 474, xid 0x61b2cbb2, Flags [none] (0x0000)
Your-IP 10.131.0.108
Client-Ethernet-Address 0a:58:0a:83:00:6c
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: ACK
Server-ID (54), length 4: 169.254.75.10
Lease-Time (51), length 4: 86313600
Default-Gateway (3), length 4: 10.131.0.1
Classless-Static-Route (121), length 5: (default:10.131.0.1)
Unknown (119), length 114: 8293,12901,11619,27765,29556,25970,29485,30777,26480,14125,25976,24941,28780,25901,13936,26224,30211,29558,25351,25452,30067,29797,29189,27759,25441,27648,883,30307,1891,27765,29556,25970,1388,28515,24940,7,25452,30067,29797,29189,27759,25441,27648,2421,29485,30565,29556,11570,1891,28525,28789,29797,2153,28276,25970,28257,27648
Hostname (12), length 19: "example-6pfpv-jjqmb"
Domain-Name (15), length 50: "e2e-clusters-x9gp7-example-6pfpv.svc.cluster.local"
Domain-Name-Server (6), length 4: 172.30.0.10
MTU (26), length 2: 8901
Subnet-Mask (1), length 4: 255.255.254.0
Also br-ex and enp1s0 appears under br-ex ovs-interface so DHCP should be retransmited Mar 30 15:12:26 example-6pfpv-jjqmb configure-ovs.sh[1205]: + nmcli conn up ovs-if-br-ex Mar 30 15:12:26 example-6pfpv-jjqmb configure-ovs.sh[1914]: + tcpdump -i any port 67 or port 68 -e -n -vv Mar 30 15:12:26 example-6pfpv-jjqmb configure-ovs.sh[1915]: + sleep 15 Mar 30 15:12:26 example-6pfpv-jjqmb configure-ovs.sh[1914]: tcpdump: data link type LINUX_SLL2 Mar 30 15:12:26 example-6pfpv-jjqmb configure-ovs.sh[1914]: dropped privs to tcpdump Mar 30 15:12:26 example-6pfpv-jjqmb configure-ovs.sh[1914]: tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1563]: + ovs-vsctl show Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1930]: 9543d237-84ff-43a0-ba00-959bc51d02d0 Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1930]: Bridge br-int Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1930]: fail_mode: secure Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1930]: datapath_type: system Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1930]: Port ovn-k8s-mp0 Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1930]: Interface ovn-k8s-mp0 Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1930]: type: internal Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1930]: Port br-int Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1930]: Interface br-int Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1930]: type: internal Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1930]: Bridge br-ex Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1930]: Port enp1s0 Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1930]: Interface enp1s0 Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1930]: type: system Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1930]: Port br-ex Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1930]: Interface br-ex Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1930]: type: internal Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1930]: ovs_version: "3.1.1" Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1563]: + ip addr Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1931]: 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1931]: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1931]: inet 127.0.0.1/8 scope host lo Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1931]: valid_lft forever preferred_lft forever Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1931]: inet6 ::1/128 scope host Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1931]: valid_lft forever preferred_lft forever Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1931]: 2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8901 qdisc fq_codel master ovs-system state UP group default qlen 1000 Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1931]: link/ether 0a:58:0a:83:00:6c brd ff:ff:ff:ff:ff:ff Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1931]: 3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1931]: link/ether d6:67:dd:2f:62:8d brd ff:ff:ff:ff:ff:ff Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1931]: 4: ovn-k8s-mp0: <BROADCAST,MULTICAST> mtu 8801 qdisc noop state DOWN group default qlen 1000 Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1931]: link/ether 62:85:1c:e0:f1:6d brd ff:ff:ff:ff:ff:ff Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1931]: 5: br-int: <BROADCAST,MULTICAST> mtu 8801 qdisc noop state DOWN group default qlen 1000 Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1931]: link/ether 16:eb:70:f0:3d:ae brd ff:ff:ff:ff:ff:ff Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1931]: 6: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8901 qdisc noqueue state UNKNOWN group default qlen 1000 Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1931]: link/ether 0a:58:0a:83:00:6c brd ff:ff:ff:ff:ff:ff Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1931]: inet6 fe80::22f5:c067:a7f7:c1d/64 scope link noprefixroute Mar 30 15:12:40 example-6pfpv-jjqmb configure-ovs.sh[1931]: valid_lft forever preferred_lft forever Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1685]: + ovs-vsctl show Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1932]: 9543d237-84ff-43a0-ba00-959bc51d02d0 Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1932]: Bridge br-int Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1932]: fail_mode: secure Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1932]: datapath_type: system Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1932]: Port ovn-k8s-mp0 Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1932]: Interface ovn-k8s-mp0 Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1932]: type: internal Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1932]: Port br-int Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1932]: Interface br-int Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1932]: type: internal Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1932]: Bridge br-ex Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1932]: Port enp1s0 Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1932]: Interface enp1s0 Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1932]: type: system Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1932]: Port br-ex Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1932]: Interface br-ex Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1932]: type: internal Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1932]: ovs_version: "3.1.1" Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1685]: + ip addr Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1933]: 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1933]: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1933]: inet 127.0.0.1/8 scope host lo Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1933]: valid_lft forever preferred_lft forever Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1933]: inet6 ::1/128 scope host Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1933]: valid_lft forever preferred_lft forever Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1933]: 2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8901 qdisc fq_codel master ovs-system state UP group default qlen 1000 Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1933]: link/ether 0a:58:0a:83:00:6c brd ff:ff:ff:ff:ff:ff Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1933]: 3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1933]: link/ether d6:67:dd:2f:62:8d brd ff:ff:ff:ff:ff:ff Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1933]: 4: ovn-k8s-mp0: <BROADCAST,MULTICAST> mtu 8801 qdisc noop state DOWN group default qlen 1000 Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1933]: link/ether 62:85:1c:e0:f1:6d brd ff:ff:ff:ff:ff:ff Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1933]: 5: br-int: <BROADCAST,MULTICAST> mtu 8801 qdisc noop state DOWN group default qlen 1000 Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1933]: link/ether 16:eb:70:f0:3d:ae brd ff:ff:ff:ff:ff:ff Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1933]: 6: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8901 qdisc noqueue state UNKNOWN group default qlen 1000 Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1933]: link/ether 0a:58:0a:83:00:6c brd ff:ff:ff:ff:ff:ff Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1933]: inet6 fe80::22f5:c067:a7f7:c1d/64 scope link noprefixroute Mar 30 15:12:41 example-6pfpv-jjqmb configure-ovs.sh[1933]: valid_lft forever preferred_lft forever Looks like known issue at OVS related to the number of threads and CPUs https://issues.redhat.com/browse/OCPBUGS-10485?focusedId=21969398&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-21969398 Can this be closed then? The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days |
Description of problem: Openshift uses the ovs-configuration script to setup the br-ex ovs-interface, this interface do a mac cloning from the primary interface and activates dhcp to take over the address, but looks like the DHCP is not send when br-ex is activated. The script also do a rollback to remove this configuration and this rollback is able to send the DHCP Version-Release number of selected component (if applicable): NetworkManager-libnm-1.42.2-1.el9.x86_64 NetworkManager-1.42.2-1.el9.x86_64 NetworkManager-team-1.42.2-1.el9.x86_64 NetworkManager-tui-1.42.2-1.el9.x86_64 NetworkManager-cloud-setup-1.42.2-1.el9.x86_64 NetworkManager-ovs-1.42.2-1.el9.x86_64 How reproducible: Always Steps to Reproduce: 1. Start a hypershift cluster with kubevirt provider 2. Run the hypershift e2e test export OCP_IMAGE_LATEST=registry.ci.openshift.org/ocp/release:4.13.0-0.nightly-2023-03-23-204038 export OCP_IMAGE_PREVIOUS=registry.ci.openshift.org/ocp/release:4.13.0-0.nightly-2023-03-23-204038 bin/test-e2e \ --test.v \ --test.timeout=0 \ --test.run='^TestCreateCluster$' \ --e2e.node-pool-replicas=1 \ --e2e.kubevirt-node-memory="6Gi" \ --e2e.platform="KubeVirt" \ --e2e.latest-release-image=${OCP_IMAGE_LATEST} \ --e2e.previous-release-image=${OCP_IMAGE_PREVIOUS} \ --e2e.pull-secret-file=/home/dvossel/pull-secret-file.txt 3. Actual results: The VM that is the worker is not ready since the br-ex ovs interface never receive the ip config from DHCP server at virt-launcher journal show that NM tries to send a DHCP discover but it never get a lease for it Mar 28 16:52:35 localhost.localdomain NetworkManager[1087]: <debug> [1680022355.8839] dhcp4 (br-ex): event: send DISCOVER to 255.255.255.255 Mar 28 16:52:36 localhost.localdomain NetworkManager[1087]: <debug> [1680022356.2804] device[c50e1b7daa6c983f] (br-ex): ip:ac6: timeout for autoconf (IPv6 router advertisement) reached Mar 28 16:52:36 localhost.localdomain NetworkManager[1087]: <debug> [1680022356.2805] device[c50e1b7daa6c983f] (br-ex): ip:ac6: set state: fail (was pending) Mar 28 16:52:36 localhost.localdomain NetworkManager[1087]: <trace> [1680022356.2806] device[c50e1b7daa6c983f] (br-ex): ip6: check-state: state pending => fail, is_failed=1, is_pending=0, is_started=1 temp_na=0, may-fail-4=0, may-fail-6=1;; manualip4=done dhcp4=pending dev4=done; manualip6=done ll6=done ac6=fail dev6=done Mar 28 16:52:36 localhost.localdomain NetworkManager[1087]: <debug> [1680022356.2806] device[c50e1b7daa6c983f] (br-ex): ip6: set state fail (was pending, reason: check-ip-state) Mar 28 16:52:36 localhost.localdomain NetworkManager[1087]: <trace> [1680022356.2806] device[c50e1b7daa6c983f] (br-ex): ip: check-state: (combined) state pending => pending Mar 28 16:52:49 localhost.localdomain NetworkManager[1087]: <trace> [1680022369.7859] settings-connection[cfbf5970fe5e8d2a,7df095d9-9373-4d28-9ff9-f0094ebf6414]: autoconnect: retries set 4 (reset) Mar 28 16:52:49 localhost.localdomain NetworkManager[1087]: <trace> [1680022369.7860] device[c50e1b7daa6c983f] (br-ex): ip:dhcp4: DHCP failing: timeout getting lease Mar 28 16:52:49 localhost.localdomain NetworkManager[1087]: <debug> [1680022369.7860] device[c50e1b7daa6c983f] (br-ex): ip:dhcp4: set state fail (was pending) So we end up with Mar 28 16:52:49 localhost.localdomain configure-ovs.sh[2096]: Error: Connection activation failed: IP configuration could not be reserved (no available address, timeout, etc.) Mar 28 16:52:49 localhost.localdomain configure-ovs.sh[2096]: Hint: use 'journalctl -xe NM_CONNECTION=7df095d9-9373-4d28-9ff9-f0094ebf6414 + NM_DEVICE=br-ex' to get more details. Expected results: The ip config should be received Additional info: This is OCP 4.13 based cluster We also see that "Network manager dispatcher service" and "hostname service" are started in the middle of "ovs-configuration" br-ex retries. The ovs-configuration systemd unit https://github.com/openshift/machine-config-operator/blob/ad89d3d5587abc0e237341b3c1e808b865308cfe/templates/common/_base/units/ovs-configuration.service.yaml#L17 The ovs-configuration script https://github.com/openshift/machine-config-operator/blob/58e4d2a2edbba81dcd12f037b9b5cffbafa9480e/templates/common/_base/files/configure-ovs-network.yaml#L2