Bug 1871032
| Summary: | MAC address changes on reboot on rebooting VM with SR-IOV interface | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Nick Satsia <nsatsia> |
| Component: | Networking | Assignee: | zenghui.shi <zshi> |
| Networking sub component: | SR-IOV | QA Contact: | zhaozhanqi <zzhao> |
| Status: | CLOSED CURRENTRELEASE | Docs Contact: | |
| Severity: | high | ||
| Priority: | medium | CC: | bbennett, zshi |
| Version: | 4.5 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.6.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-08-26 04:04:14 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
The issue might be that VF admin MAC is all-zero by default(observed via "ip link show <PF name>" on the host), so every time when VM restarts, VF gets a new MAC address instead of restoring from admin MAC address. In 4.6, SR-IOV Operator sets the admin MAC address to a non-zero value upon system(node) starts , which may fix the issue. @Nick, do we expect the MAC address to be static even after system(node) reboot? or only VM reboot? will you be able to test the same scenario with latest 4.6 SR-IOV Operator? (In reply to zenghui.shi from comment #1) > The issue might be that VF admin MAC is all-zero by default(observed via "ip > link show <PF name>" on the host), so every time when VM restarts, VF gets a > new MAC address instead of restoring from admin MAC address. > In 4.6, SR-IOV Operator sets the admin MAC address to a non-zero value upon > system(node) starts , which may fix the issue. > > @Nick, do we expect the MAC address to be static even after system(node) > reboot? or only VM reboot? > will you be able to test the same scenario with latest 4.6 SR-IOV Operator? I come from a Telco and OpenStack background so my expectation is that any MAC addresses assigned to the VM will remain static for the life of the VM, until it is deleted. Especially considering the MAC address is hard-coded in the ifcfg file on the VM so on reboot/restart they fail to bind as the MAC address is now wrong. I can test it with the 4.6 SR-IOV Operator, just need your help on how to get it. Or do I have to rebuild my cluster with the 4.6.nightly? BTW, you are correct in that the MAC is set to all 0s on the host, 3: ens6f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 40:a6:b7:15:f4:4c brd ff:ff:ff:ff:ff:ff vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 2000, tx rate 2048 (Mbps), max_tx_rate 2048Mbps, spoof checking on, link-state auto, trust off vf 1 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off vf 2 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off vf 3 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off vf 4 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 2000, tx rate 2048 (Mbps), max_tx_rate 2048Mbps, spoof checking on, link-state auto, trust off vf 5 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off (In reply to Nick Satsia from comment #2) > (In reply to zenghui.shi from comment #1) > > The issue might be that VF admin MAC is all-zero by default(observed via "ip > > link show <PF name>" on the host), so every time when VM restarts, VF gets a > > new MAC address instead of restoring from admin MAC address. > > In 4.6, SR-IOV Operator sets the admin MAC address to a non-zero value upon > > system(node) starts , which may fix the issue. > > > > @Nick, do we expect the MAC address to be static even after system(node) > > reboot? or only VM reboot? > > will you be able to test the same scenario with latest 4.6 SR-IOV Operator? > > I come from a Telco and OpenStack background so my expectation is that any > MAC addresses assigned to the VM will remain static for the life of the VM, > until it is deleted. Especially considering the MAC address is hard-coded in > the ifcfg file on the VM so on reboot/restart they fail to bind as the MAC > address is now wrong. > > I can test it with the 4.6 SR-IOV Operator, just need your help on how to > get it. > Or do I have to rebuild my cluster with the 4.6.nightly? You could try re-install the SR-IOV operator. 1) Uninstall the 4.5 SR-IOV Operator (if it's installed via subscription, unsubscribe) 2) Install latest 4.6 SR-IOV Operator follows the quick start doc: https://github.com/openshift/sriov-network-operator/blob/master/doc/quickstart.md#installation Alternative is (I assume the OCP is deployed in a baremetal environment): 1) Manually set the admin MAC address of VFs on the baremetal host via cmd "ip link set ens6f0 vf <vf-index> mac <random-mac-address>" 2) Re-create the VMs, then test with rebooting. Above two approaches set the admin MAC address before creating VMs, this makes sure that whenever libvirt wants to create the VM, it can restore the MAC address from VF admin address. This only guarantees MAC address is consistent across VM rebooting, not system rebooting because after system rebooting, VFs will be re-created and assigned with new admin MAC. > > > > BTW, you are correct in that the MAC is set to all 0s on the host, > > 3: ens6f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode > DEFAULT group default qlen 1000 > link/ether 40:a6:b7:15:f4:4c brd ff:ff:ff:ff:ff:ff > vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 2000, > tx rate 2048 (Mbps), max_tx_rate 2048Mbps, spoof checking on, link-state > auto, trust off > vf 1 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof > checking on, link-state auto, trust off > vf 2 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof > checking on, link-state auto, trust off > vf 3 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof > checking on, link-state auto, trust off > vf 4 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 2000, > tx rate 2048 (Mbps), max_tx_rate 2048Mbps, spoof checking on, link-state > auto, trust off > vf 5 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof > checking on, link-state auto, trust off (In reply to zenghui.shi from comment #3) > (In reply to Nick Satsia from comment #2) > > (In reply to zenghui.shi from comment #1) > > > The issue might be that VF admin MAC is all-zero by default(observed via "ip > > > link show <PF name>" on the host), so every time when VM restarts, VF gets a > > > new MAC address instead of restoring from admin MAC address. > > > In 4.6, SR-IOV Operator sets the admin MAC address to a non-zero value upon > > > system(node) starts , which may fix the issue. > > > > > > @Nick, do we expect the MAC address to be static even after system(node) > > > reboot? or only VM reboot? > > > will you be able to test the same scenario with latest 4.6 SR-IOV Operator? > > > > I come from a Telco and OpenStack background so my expectation is that any > > MAC addresses assigned to the VM will remain static for the life of the VM, > > until it is deleted. Especially considering the MAC address is hard-coded in > > the ifcfg file on the VM so on reboot/restart they fail to bind as the MAC > > address is now wrong. > > > > I can test it with the 4.6 SR-IOV Operator, just need your help on how to > > get it. > > Or do I have to rebuild my cluster with the 4.6.nightly? > > You could try re-install the SR-IOV operator. > 1) Uninstall the 4.5 SR-IOV Operator (if it's installed via subscription, > unsubscribe) > 2) Install latest 4.6 SR-IOV Operator follows the quick start doc: > https://github.com/openshift/sriov-network-operator/blob/master/doc/ > quickstart.md#installation > > Alternative is (I assume the OCP is deployed in a baremetal environment): > 1) Manually set the admin MAC address of VFs on the baremetal host via cmd > "ip link set ens6f0 vf <vf-index> mac <random-mac-address>" > 2) Re-create the VMs, then test with rebooting. > > Above two approaches set the admin MAC address before creating VMs, this > makes sure that whenever libvirt wants to create the VM, it can restore the > MAC address from VF admin address. > This only guarantees MAC address is consistent across VM rebooting, not > system rebooting because after system rebooting, VFs will be re-created and > assigned with new admin MAC. > > > > > > > > > BTW, you are correct in that the MAC is set to all 0s on the host, > > > > 3: ens6f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode > > DEFAULT group default qlen 1000 > > link/ether 40:a6:b7:15:f4:4c brd ff:ff:ff:ff:ff:ff > > vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 2000, > > tx rate 2048 (Mbps), max_tx_rate 2048Mbps, spoof checking on, link-state > > auto, trust off > > vf 1 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof > > checking on, link-state auto, trust off > > vf 2 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof > > checking on, link-state auto, trust off > > vf 3 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof > > checking on, link-state auto, trust off > > vf 4 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 2000, > > tx rate 2048 (Mbps), max_tx_rate 2048Mbps, spoof checking on, link-state > > auto, trust off > > vf 5 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof > > checking on, link-state auto, trust off I tested it the manual way configuring MACs on the VFs: ip link set ens6f0 vf 0 mac 00:16:3e:0a:2a:ae ip link set ens6f0 vf 1 mac 00:16:3e:19:e1:73 ip link set ens6f0 vf 2 mac 00:16:3e:74:0b:cc ip link set ens6f0 vf 3 mac 00:16:3e:45:4f:95 ip link set ens6f0 vf 4 mac 00:16:3e:32:03:19 ip link set ens6f0 vf 5 mac 00:16:3e:61:e6:00 I can confirm this fixes the issue. Thanks.
>
> I tested it the manual way configuring MACs on the VFs:
>
> ip link set ens6f0 vf 0 mac 00:16:3e:0a:2a:ae
> ip link set ens6f0 vf 1 mac 00:16:3e:19:e1:73
> ip link set ens6f0 vf 2 mac 00:16:3e:74:0b:cc
> ip link set ens6f0 vf 3 mac 00:16:3e:45:4f:95
> ip link set ens6f0 vf 4 mac 00:16:3e:32:03:19
> ip link set ens6f0 vf 5 mac 00:16:3e:61:e6:00
>
> I can confirm this fixes the issue.
>
> Thanks.
Cool, Nick, do we need to backport the fix to 4.5?
|
Description of problem: Running a Centos7 VM with CNV on OCP4.5.5 (patched to 4.5.6) with an SR-IOV multus interface. Every time I reboot the VM or stop/start it, it gets a new MAC address. This results in the ifcfg file being incorrect as the original MAC is hard-coded in it, breaking interface bring-up on reboot. Version-Release number of selected component (if applicable): OCP 4.5.5 upgraded to 4.5.6 candidate How reproducible: 100% Steps to Reproduce: 1. Create a CNV VM with an SR-IOV interface 2. Note the MAC address of the VM SR-IOV interface 3. Reboot the VM form the linux command line. 4. Note that after the reboot the SR-IOV interface has a new MAC address. Actual results: The SR-IOV interface gets a new MAC address on every reboot. Expected results: The MAC address should be constant/persistent. Additional info: 1. Using an Intel X710 NIC card. However, this should not be impacting the results. 2. Setting a static MAC address works around this issue.