the issue the letters case, kubevirt always assume that the key is 'userdata' while the IPI creates the secrets with key 'userData'.

Could you please attach the VM's yaml file?

@Danielle Also, can we get some more info about why GCP ipi is generating a secret that needs to be consumed directly by KubeVirt and why that secret can't be formatted in a way that KubeVirt supports.

I'm not sure I understand the statement in the description

"kubevirt looking for incorrect key name userdata instead of userData"

I wouldn't say that it's accurate that KubeVirt is looking in the wrong place. The secret key "userdata" is what we support for cloud-init userdata stored in a k8s secret. We can discuss also supporting "userData" as camel case, but I want to have a clear understanding for why we're doing this.

@David Vossel, I'm not sure that it's a bug, and you're right kubevirt is not looking for the wrong key but in my opinion it needs supporting also the camel case. 
This issue catch me up when I ran a POC and I tried to add more worker nodes by adding a VMs to an OCP cluster that running on GCP provider.
I saw that the other parameter names are appear with camel case ('cloudInitConfigDrive', 'dataVolume'...) and that's why I thought that it should also be with a camel case. 

The VM yaml is: 

```
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  name: worker1
spec:
  dataVolumeTemplates:
  - metadata:
      name: worker1
    spec:
      pvc:
       accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 35Gi
      source:
        pvc:
          name: rhcos-image
          namespace: test
    status: {}
  running: true
  template:
    metadata:
      creationTimestamp: null
      labels:
        kubevirt.io/vm: worker1
    spec:
      domain:
        devices:
          disks:
            - disk:
                bus: virtio
              name: worker1-datavolumedisk1
            - disk:
                bus: sata
              name: worker1-cloudinitdisk
        machine:
          type: ""
        resources:
          requests:
            memory: 4G
            cpu: 2
      volumes:
        - dataVolume:
            name: worker1-bootvolume
          name: worker1-datavolumedisk1
        - cloudInitConfigDrive:
            secretRef:
              name: worker-user-data
          name: worker1-cloudinitdisk


```

PR updated, shouldn't be so controversial anymore, just needs a review (https://github.com/kubevirt/kubevirt/pull/3929)

--------------------------------------------------------------------
[kbidarka@localhost vm-secret]$ cat my-vm-secret.yaml 
apiVersion: v1
kind: Secret
metadata:
  name: my-vm-secret
type: Opaque
data:
  userData: I2Nsb3VkLWNvbmZpZwpwYXNzd29yZDogYXRvbWljCnNzaF9wd2F1dGg6IFRydWUKY2hwYXNzd2Q6IHsgZXhwaXJlOiBGYWxzZSB9Cg==

[kbidarka@localhost vm-secret]$ oc apply -f my-vm-secret.yaml 
[kbidarka@localhost vm-secret]$ oc get secret my-vm-secret -o yaml 
apiVersion: v1
data:
  userdata: I2Nsb3VkLWNvbmZpZwpwYXNzd29yZDogYXRvbWljCnNzaF9wd2F1dGg6IFRydWUKY2hwYXNzd2Q6IHsgZXhwaXJlOiBGYWxzZSB9Cg==
kind: Secret
metadata:
  annotations:


(cnv-tests) [kbidarka@localhost vm-secret]$ echo "I2Nsb3VkLWNvbmZpZwpwYXNzd29yZDogYXRvbWljCnNzaF9wd2F1dGg6IFRydWUKY2hwYXNzd2Q6IHsgZXhwaXJlOiBGYWxzZSB9Cg==" | base64 -d
#cloud-config
password: atomic
ssh_pwauth: True
chpasswd: { expire: False }


-------------------------------------------------------------------------

-------------------------------------------------------------------

spec:
  running: true
  template:
    metadata:
      creationTimestamp: null
      labels:
        kubevirt.io/domain: vm-nfs-rhel82
        kubevirt.io/size: small
    spec:
      domain:
        cpu:
          cores: 1
          sockets: 1
          threads: 1
        devices:
          disks:
          - disk:
              bus: virtio
            name: rootdisk
          - disk:
              bus: virtio
            name: cloudinitdisk
          interfaces:
          - masquerade: {}
            name: default
          networkInterfaceMultiqueue: true
          rng: {}
        machine:
          type: pc-q35-rhel8.2.0
        resources:
          requests:
            memory: 2Gi
      evictionStrategy: LiveMigrate
      networks:
      - name: default
        pod: {}
      terminationGracePeriodSeconds: 180
      volumes:
      - name: rootdisk
        persistentVolumeClaim:
          claimName: dv-nfs-rhel82
      - cloudInitConfigDrive:
          secretRef:
            name: my-vm-secret
        name: cloudinitdisk


--------------------------------------------------------------------

(cnv-tests) [kbidarka@localhost vm-secret]$ virtctl console vm-nfs-rhel82
Successfully connected to vm-nfs-rhel82 console. The escape sequence is ^]

[cloud-user@vm-nfs-rhel82 ~]$ exit
logout

Red Hat Enterprise Linux 8.2 (Ootpa)
Kernel 4.18.0-193.19.1.el8_2.x86_64 on an x86_64

Activate the web console with: systemctl enable --now cockpit.socket

vm-nfs-rhel82 login: cloud-user
Password: 
Last login: Mon Oct 19 15:21:09 on ttyS0
[cloud-user@vm-nfs-rhel82 ~]$ 


(cnv-tests) [kbidarka@localhost vm-secret]$ oc get pods
NAME                                READY     STATUS    RESTARTS   AGE
virt-launcher-vm-nfs-rhel82-jwvr2   1/1       Running   0          13m
(cnv-tests) [kbidarka@localhost vm-secret]$ oc rsh -c compute virt-launcher-vm-nfs-rhel82-jwvr2 
sh-4.4# cd /var/run/kubevirt-private/
sh-4.4# cd secret
sh-4.4# cd cloudinitdisk/
sh-4.4# ls
userdata
sh-4.4# cat userdata 
#cloud-config
password: atomic
ssh_pwauth: True
chpasswd: { expire: False }
sh-4.4# exit
exit
-----------------------------------------------------------------------


When using above, "userdata":
    Was able to login to the VMI using the passwd "atomic" successfully, which was set in the "userdata" base64 format as shown above.

I was under the impression that, the secret should work when using "userData". 

Not 100% sure, but need someone to confirm this.

-------------------

[kbidarka@localhost vm-secret]$ oc get secret my-vm-secret -o yaml 
apiVersion: v1
data:
  userData: I2Nsb3VkLWNvbmZpZwpwYXNzd29yZDogYXRvbWljCnNzaF9wd2F1dGg6IFRydWUKY2hwYXNzd2Q6IHsgZXhwaXJlOiBGYWxzZSB9Cg==
kind: Secret
metadata:

-------------------

Using the above secret, I see the below issue:

(cnv-tests) [kbidarka@localhost vm-secret]$ virtctl start vm-nfs-rhel82
VM vm-nfs-rhel82 was scheduled to start
(cnv-tests) [kbidarka@localhost vm-secret]$ oc get pods
NAME                                READY     STATUS    RESTARTS   AGE
virt-launcher-vm-nfs-rhel82-hx2h5   1/1       Running   0          18s
(cnv-tests) [kbidarka@localhost vm-secret]$ oc rsh virt-launcher-vm-nfs-rhel82-hx2h5
sh-4.4# cd /var/run/kubevirt-private/secret/cloudinitdisk/
sh-4.4# ls
userdata
sh-4.4# cd userdata/ 
sh-4.4# ls
sh-4.4# exit
exit
(cnv-tests) [kbidarka@localhost vm-secret]$ oc logs -f  virt-launcher-vm-nfs-rhel82-hx2h5

{"component":"virt-launcher","level":"error","msg":"internal error: Child process (/usr/sbin/dmidecode -q -t 0,1,2,3,4,17) unexpected exit status 1: /dev/mem: No such file or directory","pos":"virCommandWait:2713","subcomponent":"libvirt","thread":"49","timestamp":"2020-10-19T20:18:57.259000Z"}
{"component":"virt-launcher","level":"info","msg":"Connected to libvirt daemon","pos":"libvirt.go:396","timestamp":"2020-10-19T20:18:57.697496Z"}
{"component":"virt-launcher","level":"info","msg":"Registered libvirt event notify callback","pos":"client.go:394","timestamp":"2020-10-19T20:18:57.701405Z"}
{"component":"virt-launcher","level":"info","msg":"Marked as ready","pos":"virt-launcher.go:71","timestamp":"2020-10-19T20:18:57.701707Z"}
{"component":"virt-launcher","kind":"","level":"info","msg":"Executing PreStartHook on VMI pod environment","name":"vm-nfs-rhel82","namespace":"default","pos":"manager.go:982","timestamp":"2020-10-19T20:18:58.301361Z","uid":"d29a8b14-0b36-40c9-9767-1f05e66f1f56"}
{"component":"virt-launcher","level":"error","msg":"could not read secret data from source: /var/run/kubevirt-private/secret/cloudinitdisk/userdata","pos":"cloud-init.go:207","reason":"read /var/run/kubevirt-private/secret/cloudinitdisk/userdata: is a directory","timestamp":"2020-10-19T20:18:58.301548Z"}
{"component":"virt-launcher","level":"error","msg":"could not read secret data from source: /var/run/kubevirt-private/secret/cloudinitdisk/networkdata","pos":"cloud-init.go:207","reason":"open /var/run/kubevirt-private/secret/cloudinitdisk/networkdata: no such file or directory","timestamp":"2020-10-19T20:18:58.301639Z"}
{"component":"virt-launcher","level":"error","msg":"could not read secret data from source: /var/run/kubevirt-private/secret/cloudinitdisk/networkData","pos":"cloud-init.go:207","reason":"open /var/run/kubevirt-private/secret/cloudinitdisk/networkData: no such file or directory","timestamp":"2020-10-19T20:18:58.301712Z"}
{"component":"virt-launcher","kind":"","level":"error","msg":"pre start setup for VirtualMachineInstance failed.","name":"vm-nfs-rhel82","namespace":"default","pos":"manager.go:1258","reason":"no cloud-init data-source found at volume: cloudinitdisk","timestamp":"2020-10-19T20:18:58.301748Z","uid":"d29a8b14-0b36-40c9-9767-1f05e66f1f56"}
{"component":"virt-launcher","kind":"","level":"error","msg":"Failed to sync vmi","name":"vm-nfs-rhel82","namespace":"default","pos":"server.go:161","reason":"no cloud-init data-source found at volume: cloudinitdisk","timestamp":"2020-10-19T20:18:58.301853Z","uid":"d29a8b14-0b36-40c9-9767-1f05e66f1f56"}

Also from the above virt-launcher pod logs, for some reason it is not looking for the "/var/run/kubevirt-private/secret/cloudinitdisk/userData". Should we be expecting it to look for that dir as well?

(In reply to Kedar Bidarkar from comment #6)
> --------------------------------------------------------------------
> [kbidarka@localhost vm-secret]$ cat my-vm-secret.yaml 
> apiVersion: v1
> kind: Secret
> metadata:
>   name: my-vm-secret
> type: Opaque
> data:
>   userData:
> I2Nsb3VkLWNvbmZpZwpwYXNzd29yZDogYXRvbWljCnNzaF9wd2F1dGg6IFRydWUKY2hwYXNzd2Q6I
> HsgZXhwaXJlOiBGYWxzZSB9Cg==
> 
> [kbidarka@localhost vm-secret]$ oc apply -f my-vm-secret.yaml 
> [kbidarka@localhost vm-secret]$ oc get secret my-vm-secret -o yaml 
> apiVersion: v1
> data:
>   userdata:
> I2Nsb3VkLWNvbmZpZwpwYXNzd29yZDogYXRvbWljCnNzaF9wd2F1dGg6IFRydWUKY2hwYXNzd2Q6I
> HsgZXhwaXJlOiBGYWxzZSB9Cg==
> kind: Secret
> metadata:
>   annotations:
> 
> 

I'm confused, the file says "userData" but on creation it was changed to "userdata"?
Is this maybe a copy-paste error? I just tested this on k8s and the string was still "userData" after creation.

Also later on, the virt-launcher pod is shown as running fine, when its logs show a critical error.

Please be more explicit on what works and what doesn't.

Thank you!

Assumed yes to the above.

Using the yaml from the previous comments, I was able to reproduce and fix the issue.
See https://github.com/kubevirt/kubevirt/pull/4670

To verify: follow reproduction steps in description

[kbidarka@localhost secret-gcp-ipi]$ cat my-vm-secret.yaml 
apiVersion: v1
kind: Secret
metadata:
  name: my-vm-secret
type: Opaque
data:
  userData: I2Nsb3VkLWNvbmZpZwpwYXNzd29yZDogYXRvbWljCnNzaF9wd2F1dGg6IFRydWUKY2hwYXNzd2Q6IHsgZXhwaXJlOiBGYWxzZSB9Cg==

[kbidarka@localhost secret-gcp-ipi]$ cat secretref-configdrive.yaml | grep -B 5 -A 10 ConfigDrive
      terminationGracePeriodSeconds: 0
      volumes:
      - dataVolume:
          name: rhel84-secref-dv2
        name: datavolumedisk1
      - cloudInitConfigDrive:
          secretRef:
            name: my-vm-secret
        name: cloudinitdisk



[kbidarka@localhost secret-gcp-ipi]$ virtctl console vm2-rhel84-secref
Successfully connected to vm2-rhel84-secref console. The escape sequence is ^]

Red Hat Enterprise Linux 8.4 Beta (Ootpa)
Kernel 4.18.0-287.el8.dt4.x86_64 on an x86_64

Activate the web console with: systemctl enable --now cockpit.socket

vm2-rhel84-secref login: cloud-user
Password: 
[root@vm2-rhel84-secref ~]# (cnv-tests) [kbidarka@localhost secret-gcp-ipi]$ 
[kbidarka@localhost secret-gcp-ipi]$ oc get pods
NAME                                    READY   STATUS      RESTARTS   AGE
virt-launcher-vm2-rhel84-6c9dr          0/1     Completed   0          44h
virt-launcher-vm2-rhel84-m26dv          1/1     Running     0          51m
virt-launcher-vm2-rhel84-secref-sl94h   1/1     Running     0          3m12s

[kbidarka@localhost secret-gcp-ipi]$ oc rsh virt-launcher-vm2-rhel84-secref-sl94h 
sh-4.4# cd /var/run/kubevirt-private/secret/cloudinitdisk/

sh-4.4# ls
userData  userdata
sh-4.4# cat userData 
#cloud-config
password: atomic
ssh_pwauth: True
chpasswd: { expire: False }


Summary: Was able to successfully login to the VM with secret using "userData".

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.8.0 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2920