As per the cloud init documentation - [0] - the only data source that allows for network config v2 is the no cloud data source. That is the configuration format you are indeed using.

For the config drive data source, you would need to use either ENI - [1] - or OpenStack Metadata Service Network - [2].

[0] - https://cloudinit.readthedocs.io/en/latest/topics/network-config.html#network-configuration-sources
[1] - https://cloudinit.readthedocs.io/en/latest/topics/network-config-format-eni.html#network-config-eni
[2] - https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/metadata-service-network-info.html#rest-api-impact

I did not mention that I really do not expect OpenStack Metadata Service Network format to work at all, since it refers to entities on a data model that does not exist on kubevirt.

https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/metadata-service-network-info.html#rest-api-impact

(In reply to Miguel Duarte Barroso from comment #3)
> I did not mention that I really do not expect OpenStack Metadata Service
> Network format to work at all, since it refers to entities on a data model
> that does not exist on kubevirt.
> 
> https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/
> metadata-service-network-info.html#rest-api-impact

OK it does work. *Some* of the attributes (vif_id, id, network_id) used make no sense on non-openstack providers but are required according to the schema [0].

Please try to repeat your experiment but with this following network data section:

```yaml
---
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  creationTimestamp: null
  labels:
    kubevirt.io/vm: vmb
  name: vmb-1616069337-0921965
spec:
  running: false
  template:
    metadata:
      creationTimestamp: null
      labels:
        kubevirt.io/domain: vmb-1616069337-0921965
        kubevirt.io/vm: vmb-1616069337-0921965
    spec:
      accessCredentials:
      - sshPublicKey:
          propagationMethod:
            configDrive:
              name: id_rsa.pub
          source:
            secret:
              secretName: my-key  # the name of the secret object ( I exported it w/ `kubectl create secret generic my-key --from-file=ssh-publickey=<path_to_public_key>` )
      domain:
        cpu:
          cores: 1
        devices:
          disks:
          - disk:
              bus: virtio
            name: containerdisk
          - disk:
              bus: virtio
            name: cloudinitdisk
          interfaces:
          - masquerade: {}
            name: default
          rng: {}
        machine:
          type: ''
        resources:
          requests:
            memory: 1024Mi
      networks:
      - name: default
        pod: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - containerDisk:
          image: quay.io/openshift-cnv/qe-cnv-tests-fedora:33
        name: containerdisk
      - cloudInitConfigDrive:
          networkData: |
            {
              "links": [
                          {
                              "id": "tap0",
                              "name": "eth0",
                              "mtu": 1500,
                              "type": "tap",
                              "vif_id": "3c54ff58-02b5-499e-bed0-03530f89e624" # pointless attribute.
                          }
                    ],
                    "networks": [
                        {
                            "id": "network0",
                            "name": "eth0",
                            "link": "tap0",
                            "network_id": "764d0ecb-f8a5-47d9-b034-53b5b61666a7", # pointless attribute.
                            "type": "ipv4_dhcp"
                        },
                        {
                            "id": "network1",
                            "name": "eth0",
                            "link": "tap0",
                            "network_id": "764d0ecb-f8a5-47d9-b034-53b5b61666a8", # pointless attribute.
                            "routes": [
                                {
                                    "gateway": "fd10:0:2::1",
                                    "netmask": "120",
                                    "network": "::0"
                                }
                            ],
                            "address": "fd10:0:2::2/120",
                            "type": "static"
                        }

                    ],
                    "services": []
                  }
          userData: |-
            #cloud-config
            chpasswd: { expire: False }
            password: fedora
        name: cloudinitdisk


```

Please confirm this works for you, so we can close the bug (not a bug).

[0] - https://docs.openstack.org/nova/latest/_downloads/9119ca7ac90aa2990e762c08baea3a36/network_data.json

@David, the issue in this bug was found when testing https://kubevirt.io/user-guide/virtual_machines/accessing_virtual_machines/#static-ssh-key-injection-via-cloud-init
Please see https://bugzilla.redhat.com/show_bug.cgi?id=1941470#c4, it appears to be the only way to use SSH injection using cloudInitConfigDrive provider and configure networkData (though the are some issues which Miguel is looking into).
Could a different provider be used? If not then I think we should change this bug to a doc bug. WDYT? 



@Miguel, tested with the provided yaml:
IP and route are set:
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:83:6a:35 brd ff:ff:ff:ff:ff:ff
    altname enp1s0
    inet 10.0.2.2/24 brd 10.0.2.255 scope global dynamic noprefixroute eth0
       valid_lft 86309432sec preferred_lft 86309432sec
    inet6 fd10:0:2::2/120 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe83:6a35/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

$ ip -6 r
::1 dev lo proto kernel metric 256 pref medium
::/120 via fd10:0:2::1 dev eth0 proto static metric 100 pref medium
fd10:0:2::/120 dev eth0 proto kernel metric 100 pref medium
fe80::/64 dev eth0 proto kernel metric 100 pref medium


MTU is not set:
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:83:6a:35 brd ff:ff:ff:ff:ff:ff
    altname enp1s0


Ping worked on your cluster but not on mine:
$ tracepath fd10:0:2::1
 1?: [LOCALHOST]                        0.032ms pmtu 1450
 1:  vmb-1616069337-0921965                              3096.493ms !H
     Resume: pmtu 1450

I think the cnv cluster is not configured in dual stack mode.

As can be seen, the launcher pods do not have an IPv6 global unicast address on their pod primary interface, which is the criteria for kubevirt to install the flows that implement DNAT/masquerade in the pod, which would enable the VM to ping the gateway.

$ kubectl get pods                                           
NAME                                        READY   STATUS      RESTARTS   AGE
...
virt-launcher-vm-mdb-bcmqb                  2/2     Running     0          7m28s
...

$ kubectl exec -it virt-launcher-vm-mdb-bcmqb -- ip addr show eth0 
Defaulting container name to compute.
Use 'kubectl describe pod/virt-launcher-vm-mdb-bcmqb -n default' to see all of the containers in this pod.
3: eth0@if680: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether 0a:58:0a:81:02:ba brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.129.2.186/23 brd 10.129.3.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::8827:2cff:fe90:633e/64 scope link 
       valid_lft forever preferred_lft forever

Are you sure the cluster is configured in dual-stack mode ? If not, there's no way the gateway will be pinged.

I have left this running VM / pod on the cluster whose credentials @rnetser gave over private msg.

(In reply to Ruth Netser from comment #5)
> @David, the issue in this bug was found when testing
> https://kubevirt.io/user-guide/virtual_machines/accessing_virtual_machines/
> #static-ssh-key-injection-via-cloud-init
> Please see https://bugzilla.redhat.com/show_bug.cgi?id=1941470#c4, it
> appears to be the only way to use SSH injection using cloudInitConfigDrive
> provider and configure networkData (though the are some issues which Miguel
> is looking into).
> Could a different provider be used?

unfortunately a different cloud-init provider can not be used for the access credentials api. ConfigDrive allows for ssh keys to be injected using metadata, where as noCloud appears to not allow this. We at least only support it with COnfigDrive. 

> If not then I think we should change
> this bug to a doc bug. WDYT? 

possibly, I'll let the network team confirm whether this is something that can be handled within docs or not.

(In reply to David Vossel from comment #8)
> (In reply to Ruth Netser from comment #5)
> > @David, the issue in this bug was found when testing
> > https://kubevirt.io/user-guide/virtual_machines/accessing_virtual_machines/
> > #static-ssh-key-injection-via-cloud-init
> > Please see https://bugzilla.redhat.com/show_bug.cgi?id=1941470#c4, it
> > appears to be the only way to use SSH injection using cloudInitConfigDrive
> > provider and configure networkData (though the are some issues which Miguel
> > is looking into).
> > Could a different provider be used?
> 
> unfortunately a different cloud-init provider can not be used for the access
> credentials api. ConfigDrive allows for ssh keys to be injected using
> metadata, where as noCloud appears to not allow this. We at least only
> support it with COnfigDrive. 
> 
> > If not then I think we should change
> > this bug to a doc bug. WDYT? 
> 
> possibly, I'll let the network team confirm whether this is something that
> can be handled within docs or not.

We could definitely update the docs indicating the special considerations to take when we want to inject ssh public keys along w/ IPv6 network configuration in the docs,
but personally, I think it raises a lot of questions; having to provision the nonsense ID related attributes to comply w/ the json schema is particularly jaw dropping.

I have investigated the cloud init automatic translation mechanisms - [0] - (imo, this would be a neat solution, since the user would just use a tool to translate our
example to the openstack metadata format) but it doesn't work in that direction (it **does work** from openstack metadata to netplan (network config v2; we need the
opposite).

I am unsure what's worse; document this, or leave it undocumented, and hope no one requires this use case.

I would say the most sensible thing to do is to discourage this use case (if you use network data, use NoCloud datasource) on the docs.

[0] - https://cloudinit.readthedocs.io/en/latest/topics/network-config.html#network-configuration-tools

(In reply to Miguel Duarte Barroso from comment #9)
> (In reply to David Vossel from comment #8)
> > (In reply to Ruth Netser from comment #5)
> > > @David, the issue in this bug was found when testing
> > > https://kubevirt.io/user-guide/virtual_machines/accessing_virtual_machines/
> > > #static-ssh-key-injection-via-cloud-init
> > > Please see https://bugzilla.redhat.com/show_bug.cgi?id=1941470#c4, it
> > > appears to be the only way to use SSH injection using cloudInitConfigDrive
> > > provider and configure networkData (though the are some issues which Miguel
> > > is looking into).
> > > Could a different provider be used?
> > 
> > unfortunately a different cloud-init provider can not be used for the access
> > credentials api. ConfigDrive allows for ssh keys to be injected using
> > metadata, where as noCloud appears to not allow this. We at least only
> > support it with COnfigDrive. 
> > 
> > > If not then I think we should change
> > > this bug to a doc bug. WDYT? 
> > 
> > possibly, I'll let the network team confirm whether this is something that
> > can be handled within docs or not.
> 
> We could definitely update the docs indicating the special considerations to
> take when we want to inject ssh public keys along w/ IPv6 network
> configuration in the docs,
> but personally, I think it raises a lot of questions; having to provision
> the nonsense ID related attributes to comply w/ the json schema is
> particularly jaw dropping.
> 
> I have investigated the cloud init automatic translation mechanisms - [0] -
> (imo, this would be a neat solution, since the user would just use a tool to
> translate our
> example to the openstack metadata format) but it doesn't work in that
> direction (it **does work** from openstack metadata to netplan (network
> config v2; we need the
> opposite).
> 
> I am unsure what's worse; document this, or leave it undocumented, and hope
> no one requires this use case.
> 
> I would say the most sensible thing to do is to discourage this use case (if
> you use network data, use NoCloud datasource) on the docs.

The issue is that you cannot use both NoCloud and cloudInitConfigDrive so I you'd like to use SSH injection, you'd have to use cloudInitConfigDrive.
> 
> [0] -
> https://cloudinit.readthedocs.io/en/latest/topics/network-config.
> html#network-configuration-tools

(In reply to Miguel Duarte Barroso from comment #9)
> (In reply to David Vossel from comment #8)
> > (In reply to Ruth Netser from comment #5)
> > > @David, the issue in this bug was found when testing
> > > https://kubevirt.io/user-guide/virtual_machines/accessing_virtual_machines/
> > > #static-ssh-key-injection-via-cloud-init
> > > Please see https://bugzilla.redhat.com/show_bug.cgi?id=1941470#c4, it
> > > appears to be the only way to use SSH injection using cloudInitConfigDrive
> > > provider and configure networkData (though the are some issues which Miguel
> > > is looking into).
> > > Could a different provider be used?
> > 
> > unfortunately a different cloud-init provider can not be used for the access
> > credentials api. ConfigDrive allows for ssh keys to be injected using
> > metadata, where as noCloud appears to not allow this. We at least only
> > support it with COnfigDrive. 
> > 
> > > If not then I think we should change
> > > this bug to a doc bug. WDYT? 
> > 
> > possibly, I'll let the network team confirm whether this is something that
> > can be handled within docs or not.
> 
> We could definitely update the docs indicating the special considerations to
> take when we want to inject ssh public keys along w/ IPv6 network
> configuration in the docs,
> but personally, I think it raises a lot of questions; having to provision
> the nonsense ID related attributes to comply w/ the json schema is
> particularly jaw dropping.
> 
> I have investigated the cloud init automatic translation mechanisms - [0] -
> (imo, this would be a neat solution, since the user would just use a tool to
> translate our
> example to the openstack metadata format) but it doesn't work in that
> direction (it **does work** from openstack metadata to netplan (network
> config v2; we need the
> opposite).
> 
> I am unsure what's worse; document this, or leave it undocumented, and hope
> no one requires this use case.
> 
> I would say the most sensible thing to do is to discourage this use case (if
> you use network data, use NoCloud datasource) on the docs.

The issue is that you cannot use both NoCloud and cloudInitConfigDrive so If you'd like to use SSH injection, you'd have to use cloudInitConfigDrive.
> 
> [0] -
> https://cloudinit.readthedocs.io/en/latest/topics/network-config.
> html#network-configuration-tools

(In reply to Ruth Netser from comment #11)
> (In reply to Miguel Duarte Barroso from comment #9)
> > (In reply to David Vossel from comment #8)
> > > (In reply to Ruth Netser from comment #5)
> > > > @David, the issue in this bug was found when testing
> > > > https://kubevirt.io/user-guide/virtual_machines/accessing_virtual_machines/
> > > > #static-ssh-key-injection-via-cloud-init
> > > > Please see https://bugzilla.redhat.com/show_bug.cgi?id=1941470#c4, it
> > > > appears to be the only way to use SSH injection using cloudInitConfigDrive
> > > > provider and configure networkData (though the are some issues which Miguel
> > > > is looking into).
> > > > Could a different provider be used?
> > > 
> > > unfortunately a different cloud-init provider can not be used for the access
> > > credentials api. ConfigDrive allows for ssh keys to be injected using
> > > metadata, where as noCloud appears to not allow this. We at least only
> > > support it with COnfigDrive. 
> > > 
> > > > If not then I think we should change
> > > > this bug to a doc bug. WDYT? 
> > > 
> > > possibly, I'll let the network team confirm whether this is something that
> > > can be handled within docs or not.
> > 
> > We could definitely update the docs indicating the special considerations to
> > take when we want to inject ssh public keys along w/ IPv6 network
> > configuration in the docs,
> > but personally, I think it raises a lot of questions; having to provision
> > the nonsense ID related attributes to comply w/ the json schema is
> > particularly jaw dropping.
> > 
> > I have investigated the cloud init automatic translation mechanisms - [0] -
> > (imo, this would be a neat solution, since the user would just use a tool to
> > translate our
> > example to the openstack metadata format) but it doesn't work in that
> > direction (it **does work** from openstack metadata to netplan (network
> > config v2; we need the
> > opposite).
> > 
> > I am unsure what's worse; document this, or leave it undocumented, and hope
> > no one requires this use case.
> > 
> > I would say the most sensible thing to do is to discourage this use case (if
> > you use network data, use NoCloud datasource) on the docs.
> 
> The issue is that you cannot use both NoCloud and cloudInitConfigDrive so If
> you'd like to use SSH injection, you'd have to use cloudInitConfigDrive.

I'm aware. 

What I'm saying is that using a config drive network data requires to use openstack metadata format,
something which is very confusing to use outside non openstack providers. 

I am recommending not documenting that, because it does raise a lot more questions than it answers
(complying with the schema requires using some attributes that quite simply just do not have any
meaning in kubevirt's data model).

Since the networkData is really only needed for IPv6 configurations, I think keeping the documentation
as is, is the best option available.

People that want to use this use case for IPv6 should either be discouraged, or explained they are
forced to use openstack metadata format, which, imo, makes no sense outside of openstack.

> > 
> > [0] -
> > https://cloudinit.readthedocs.io/en/latest/topics/network-config.
> > html#network-configuration-tools

(In reply to Miguel Duarte Barroso from comment #12)
> (In reply to Ruth Netser from comment #11)
> > (In reply to Miguel Duarte Barroso from comment #9)
> > > (In reply to David Vossel from comment #8)
> > > > (In reply to Ruth Netser from comment #5)
> > > > > @David, the issue in this bug was found when testing
> > > > > https://kubevirt.io/user-guide/virtual_machines/accessing_virtual_machines/
> > > > > #static-ssh-key-injection-via-cloud-init
> > > > > Please see https://bugzilla.redhat.com/show_bug.cgi?id=1941470#c4, it
> > > > > appears to be the only way to use SSH injection using cloudInitConfigDrive
> > > > > provider and configure networkData (though the are some issues which Miguel
> > > > > is looking into).
> > > > > Could a different provider be used?
> > > > 
> > > > unfortunately a different cloud-init provider can not be used for the access
> > > > credentials api. ConfigDrive allows for ssh keys to be injected using
> > > > metadata, where as noCloud appears to not allow this. We at least only
> > > > support it with COnfigDrive. 
> > > > 
> > > > > If not then I think we should change
> > > > > this bug to a doc bug. WDYT? 
> > > > 
> > > > possibly, I'll let the network team confirm whether this is something that
> > > > can be handled within docs or not.
> > > 
> > > We could definitely update the docs indicating the special considerations to
> > > take when we want to inject ssh public keys along w/ IPv6 network
> > > configuration in the docs,
> > > but personally, I think it raises a lot of questions; having to provision
> > > the nonsense ID related attributes to comply w/ the json schema is
> > > particularly jaw dropping.
> > > 
> > > I have investigated the cloud init automatic translation mechanisms - [0] -
> > > (imo, this would be a neat solution, since the user would just use a tool to
> > > translate our
> > > example to the openstack metadata format) but it doesn't work in that
> > > direction (it **does work** from openstack metadata to netplan (network
> > > config v2; we need the
> > > opposite).
> > > 
> > > I am unsure what's worse; document this, or leave it undocumented, and hope
> > > no one requires this use case.
> > > 
> > > I would say the most sensible thing to do is to discourage this use case (if
> > > you use network data, use NoCloud datasource) on the docs.
> > 
> > The issue is that you cannot use both NoCloud and cloudInitConfigDrive so If
> > you'd like to use SSH injection, you'd have to use cloudInitConfigDrive.
> 
> I'm aware. 
> 
> What I'm saying is that using a config drive network data requires to use
> openstack metadata format,
> something which is very confusing to use outside non openstack providers. 
> 
> I am recommending not documenting that, because it does raise a lot more
> questions than it answers
> (complying with the schema requires using some attributes that quite simply
> just do not have any
> meaning in kubevirt's data model).

This is exactly why I think we should have this issue documented along with the SSH injection feature :)
@Petr, wdyt?

> 
> Since the networkData is really only needed for IPv6 configurations, I think
> keeping the documentation
> as is, is the best option available.
> 
> People that want to use this use case for IPv6 should either be discouraged,
> or explained they are
> forced to use openstack metadata format, which, imo, makes no sense outside
> of openstack.
> 
> > > 
> > > [0] -
> > > https://cloudinit.readthedocs.io/en/latest/topics/network-config.
> > > html#network-configuration-tools

I tend to agree that this should remain open. We should not document anything about the OpenStack network format but at the same time, it should not be blocked.

For the documentation, I would do it the other way around - whenever we document networkData, we should say that is is supported only with noCloud. So we don't need to document it on every place where we use cloudInitConfigDrive. Would that make sense?

> For the documentation, I would do it the other way around - whenever we document networkData, we should say that is is supported only with noCloud. So we don't need to document it on every place where we use cloudInitConfigDrive. Would that make sense?

Do you think it would be reasonable to create a knowledge base article for ipv6+configDrive to at least close the gap a bit here?

(In reply to David Vossel from comment #15)
> > For the documentation, I would do it the other way around - whenever we document networkData, we should say that is is supported only with noCloud. So we don't need to document it on every place where we use cloudInitConfigDrive. Would that make sense?
As a user may bump into this when trying to define networkData, I lean toward documenting it there.
> 
> Do you think it would be reasonable to create a knowledge base article for
> ipv6+configDrive to at least close the gap a bit here?

@Petr?

(In reply to Ruth Netser from comment #16)
> (In reply to David Vossel from comment #15)
> > > For the documentation, I would do it the other way around - whenever we document networkData, we should say that is is supported only with noCloud. So we don't need to document it on every place where we use cloudInitConfigDrive. Would that make sense?
> As a user may bump into this when trying to define networkData, I lean
> toward documenting it there.

Just to be sure, by there you mean in suggested networkData documentation (in this case, wherever we document how to use networkData with dual-stack)?

> > 
> > Do you think it would be reasonable to create a knowledge base article for
> > ipv6+configDrive to at least close the gap a bit here?
> 
> @Petr?

I would be for a knowledge base, but against documenting the awkward API there. Could we instead state that "networkData is not supported with cloudInitConfigDrive" and "in case you need to perform additional network configuration, use the userData section and a command line network configuration tool depending on your guest OS" and below that, show an example of how to do this with NetworkManager. This way it would be clear, within borders of what we want to support and still helpful in a situation when somebody wants to use both of those features. Thoughts?

Sounds good to me.

Proposed https://access.redhat.com/articles/6022791. 

@rnetser Could you review it ? 

moving the bug to post.

@mduarted, maybe add a note that a service needs to be created to access the VM via SSH (not directly related to the topic but relevant for SSH)?
Other then that, LGTM

(In reply to Ruth Netser from comment #20)
> @mduarted, maybe add a note that a service needs to be created to
> access the VM via SSH (not directly related to the topic but relevant for
> SSH)?

I am not specifying any port in the spec, which would make all ports reachable. 

Never the less, access to the VM via SSH is not in scope of the article (but I think the provided example does not prevent it). 

> Other then that, LGTM

A workaround was provided via a KBase article that should be published once both dual-stack and SSH key injection features get published.