Created attachment 1651849 [details] bridge policy Description of problem: When adding a secondary NIC via the web UI console, the actual result is that the default interface is modified and no new NIC is added. Version-Release number of selected component (if applicable): OCP version: 4.3.0-0.nightly-2020-01-11-070223 CNV version: 2.2.0 How reproducible: Always Steps to Reproduce: 1. Create a bridge (e.g. linux-bridge) on an existing network interface on the cluster's nodes. You can use the attached NodeNetworkConfigurationPolicy spec yaml by running: $ oc create -f nncp-br.yaml 2. Create a NetworkAttachmentDefinition using this bridge. You can use the attached NetworkAttachmentDefinition spec yaml by running: $ oc create -f br-nad.yaml 3. Create a simple VM. You can use the attached VM spec yaml by running: $ oc create -f vm-fedora.yaml 4. Start the VM by running $ virtctl start vm-fedora 5. Once the the VM is up in "Running" state - open a web console session and access the VM by going to Home -> Workloads -> Virtual Machines -> vm-fedora 6. Add a secondary NIC: a. Go to Network Interfaces tab. b. Press "Create Network Interface" button. c. Fill all required fields: - Add a name to the secondary NIC - Select Model (I selected VirtIO) - Select the NetworkAttachmentDefinition you created earlier (br11-conf) from the Network drop-down list. - Select "bridge" from the Type drop-down list. Leave the MAC address field empty. d. Press "Add" button. 7. Restart the VM (Actions -> Restart Virtual Machine) 8. After the VM is back in Running state - access its console by running: $ virtctl console vm-fedora 9. View the VM's network interfaces: $ ip link Actual results: <BUG> Only the default interface (eth0) appears, and the new secondary interface does not. Expected results: The new interface should exist in addition to the default interface. Additional info: When viewing the VM's spec yaml *after* adding the secondary interface and restart - this is what exists under interfaces and under networks, so we can see that what actually happens is that the default NIC is being modified: interfaces: - bridge: {} macAddress: 02:91:e8:00:00:01 model: virtio name: sec1 networks: - multus: networkName: br11-conf name: sec1
Created attachment 1651850 [details] NetworkAttachmentDefinition
Created attachment 1651851 [details] VM spec yaml
I cant reproduce this on the 4.3 nor on the current master. Do you have an env on which this reproduces? Can you please also provide logs from the javascript console when this happens? (e.g. in the browser the logs from javascript console). And, if you have an env on which this happens, could you please off-list provide access to it so we can debug?
Contact me via google chat and we can reproduce it together on my env. I'm attaching the log from the web console (web-console.log) I think I have a lead on the source of the bug: - After the VM is created (and started), and *before adding a secondary NIC*, the VM spec yaml doesn't include any lists of neither "interfaces" nor "networks" (see attached vm-before-add-nic.yaml). The default interface is set-up from the VM image. - When adding a secondary NIC, new "interfaces" and "networks" lists are added to the spec. These lists should include the default network and interface, and the new secondary NIC (which was added by the UI) should be added as *another* entry in both "networks" and "interfaces" lists. I suspect that what actually happens is that the new secondary NIC is added as the *first* entry of these lists.
Created attachment 1652693 [details] VM spec before adding a secondary NIC
Created attachment 1652694 [details] Web console log
ok, this is an interesting one. I would say it behaves as designed both on frontend and on backend but the combination makes the result very confusing. This is what happens: 1: create a VM which does not have any network interfaces defined 2: look at the VM yaml or to the UI's NICs tab, both properly show that there are no NICs 3: start the VM 4: the backend decides that since the VM yaml does not contain any NIC defined, it will add one. So the backend will create a VMI which is different than the VM (adds one NIC to it) 5: now go to the UI's NICs tab and check - it will tell you that the VM does not have any NICs since the VM yaml does not contain any. 6: add a new NIC from the UI. The UI properly adds a new nic into the VM yaml 7: now you have a VM yaml with one nic and a VMI yaml with a different one 8: stop the VM 9: the VMI is deleted and you are left with only the VM which contains the one NIC defined in UI 10: now start the VM - the backend will properly copy the NIC configuration from VM to VMI and will not touch it this time So it looks like the UI replaced the NIC instead of adding a new one. Even though it is not quite true. My first question would be, why is the backend adding a NIC to a VM if the user did not ask for it? Is there some reason for that logic or can it be removed?
> My first question would be, why is the backend adding a NIC to a VM if the user did not ask for it? Is there some reason for that logic or can it be removed? The answer is that this NIC is the default VMI NIC (also referred to as "pod interface"). It must exist on the VMI, therefore it is correct behavior to have it in the VMI yaml. A suggested solution might be to update the VM spec yaml once the VMI is updated with this default interface. Anyway - given Tomas's resolution that once starting the VM, the VMI spec yaml includes the default NIC while the VM yaml does not (which I have just confirmed on my cluster), then I think that this is not a UI bug, right?
(In reply to Yossi Segev from comment #8) > > My first question would be, why is the backend adding a NIC to a VM if the user did not ask for it? Is there some reason for that logic or can it be removed? > > The answer is that this NIC is the default VMI NIC (also referred to as "pod > interface"). It must exist on the VMI, therefore it is correct behavior to > have it in the VMI yaml. I actually question if it has to be on the VMI. I don't think there is a technical reason why the user would not be able to start a VM with no NIC attached to it. It is just a default behavior of kubevirt that it adds the NIC there if the user did not specify it. But it could as well not do it. > A suggested solution might be to update the VM spec yaml once the VMI is > updated with this default interface. that would be a very non-kubernetes way of doing it. The system should not modify my configuration once I create it. > > Anyway - given Tomas's resolution that once starting the VM, the VMI spec > yaml includes the default NIC while the VM yaml does not (which I have just > confirmed on my cluster), then I think that this is not a UI bug, right? Indeed. I was just talking to Petr and he asked me to move it to network. So, here it goes :)
Thanks a lot, both of you. Those are some interesting findings. Yes, the API is little weird in this, especially the defaulting. However, I believe that this is not a critical bug for 2.2, I'm moving it to "future". I hope we will be able to make this easier to grasp for users. CC Fabian, I think this is something we should look into in parallel to the Pod network rework effort. The fact that we attach VM to the default network when no network/interface is specified may be comfortable for the most basic use-case, but it also makes any other network configuration more difficult than it should be.
Good point, Petr. Please share the VM and VMI yamls _after_ the secondary NIC got attached. But my assumption right now is: a. Initial VMI does not have any vNIC defined, thus KubeVirt defaults to one attached ot pod network. b. Once a vNIC is added (connected to the bridge), a vNIC is defined, and thus KubeVir does not create one. In both cases there are different vNICs but in total only one. For consistency I wonder if we should always add the pod network. Thus in case b you would have ended up with 2 vNICS (pod and bridge). The issue is that this is maybe confusing the user as well. The other option is to flag all templates with autoattachPodInterface: false (https://kubevirt.io/api-reference/master/definitions.html#_v1_devices) in order to avoid that the pod network is auto attached. But this would not have helped here, as this bug is a bout a custom written VM. In general there is value to auto attach to pod interface and I i.e. do not see it as an option to not attach to it by default.
> Please share the VM and VMI yamls _after_ the secondary NIC got attached. I'm not sure whether you need the specs of the current, buggy case, when the default VM doesn't include default NIC, or rather the (desired?) case where the VM does specify the default NIC, and any additional NIC is *added* to the default - so I pasting both cases. Please note that I'm pasting only the interfaces and networks sections. They are similar in both VM and VMI specs: 1. Current case - Secondary NIC attached after default NIC didn't exist in VM: interfaces: - bridge: {} macAddress: 02:65:dd:00:00:06 model: virtio name: sec1 networks: - multus: networkName: br11-conf name: sec1 2. Secondary NIC attached after default NIC exists in both VM and VMI: interfaces: - bridge: {} macAddress: 02:65:dd:00:00:04 name: default - bridge: {} macAddress: 02:65:dd:00:00:05 model: virtio name: sec1 networks: - name: default pod: {} - multus: networkName: br11-conf name: sec1
Roman, do you have thoughts on this matter?
It is indeed a little bit confusing that we only add the pod network if nothing is there, and don't add it as soon as one entry is there. We have a `autoattachPodNetwork` boolean too. However forcing people to set this boolean when they add a network by hand is also strange. I think that the most "consistent" but indeed not fully optimal solution which we have is: "If you don't specify networks we will add the default, if you specify some we don't add defaults".
I would suggest in the UI case that, if the default pod network is expected to be present, to really explicitly add it. I think for UIs it is always the savest thing to set what matches the UI, and not rely too much on server defaults.
And now I fully get the issue. It is indeed tricky. We simply don't know if the VM ever ran before if it was custom created. However, I think that the UI still "knows" that when adding the interface via the UI, that before that it had no interface and it should therefore have been rendered in the UI with the pod network attached. So the user sees in the UI first the default pod network, and when adding the second nic, both could be added explicitly now. However, if it is well displayed to the user that this "Add" replaces the default network, so that the user can take action in the dialogue, it would also be possible that by default we only replace the implicit default. I suppose it can be handled with a little bit of care in the UI dialogues.
Thanks Roman. Tomas, does latest Roman's comment sound doable?
Well, in theory it could, but it would bring a lot of complexity into the UI. Also, the UI would be "guessing" that the backend will ultimately add such an interface so it would render it as if it would be there. But if the backend decides that it does not want to have that hardcoded nic there, the UI would have to be adjusted. So, long story short, there is no API we could query, only understanding of the internals of the backend and hoping it wont change so we can mimic it in the UI.
Thanks Tomáš. Seems like we don't have a reasonable way to implement this on UI and API of KubeVirt is unlikely to change. I don't see any way out of this. Marking this as WONTFIX. If anybody has a good idea how to fix this issue, feel free to reopen.