Bug 1862997

the issue the letters case, kubevirt always assume that the key is 'userdata' while the IPI creates the secrets with key 'userData'.

Could you please attach the VM's yaml file?

@Danielle Also, can we get some more info about why GCP ipi is generating a secret that needs to be consumed directly by KubeVirt and why that secret can't be formatted in a way that KubeVirt supports.

I'm not sure I understand the statement in the description

"kubevirt looking for incorrect key name userdata instead of userData"

I wouldn't say that it's accurate that KubeVirt is looking in the wrong place. The secret key "userdata" is what we support for cloud-init userdata stored in a k8s secret. We can discuss also supporting "userData" as camel case, but I want to have a clear understanding for why we're doing this.

@David Vossel, I'm not sure that it's a bug, and you're right kubevirt is not looking for the wrong key but in my opinion it needs supporting also the camel case. 
This issue catch me up when I ran a POC and I tried to add more worker nodes by adding a VMs to an OCP cluster that running on GCP provider.
I saw that the other parameter names are appear with camel case ('cloudInitConfigDrive', 'dataVolume'...) and that's why I thought that it should also be with a camel case. 

The VM yaml is: 

```
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  name: worker1
spec:
  dataVolumeTemplates:
  - metadata:
      name: worker1
    spec:
      pvc:
       accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 35Gi
      source:
        pvc:
          name: rhcos-image
          namespace: test
    status: {}
  running: true
  template:
    metadata:
      creationTimestamp: null
      labels:
        kubevirt.io/vm: worker1
    spec:
      domain:
        devices:
          disks:
            - disk:
                bus: virtio
              name: worker1-datavolumedisk1
            - disk:
                bus: sata
              name: worker1-cloudinitdisk
        machine:
          type: ""
        resources:
          requests:
            memory: 4G
            cpu: 2
      volumes:
        - dataVolume:
            name: worker1-bootvolume
          name: worker1-datavolumedisk1
        - cloudInitConfigDrive:
            secretRef:
              name: worker-user-data
          name: worker1-cloudinitdisk


```

PR updated, shouldn't be so controversial anymore, just needs a review (https://github.com/kubevirt/kubevirt/pull/3929)

--------------------------------------------------------------------
[kbidarka@localhost vm-secret]$ cat my-vm-secret.yaml 
apiVersion: v1
kind: Secret
metadata:
  name: my-vm-secret
type: Opaque
data:
  userData: I2Nsb3VkLWNvbmZpZwpwYXNzd29yZDogYXRvbWljCnNzaF9wd2F1dGg6IFRydWUKY2hwYXNzd2Q6IHsgZXhwaXJlOiBGYWxzZSB9Cg==

[kbidarka@localhost vm-secret]$ oc apply -f my-vm-secret.yaml 
[kbidarka@localhost vm-secret]$ oc get secret my-vm-secret -o yaml 
apiVersion: v1
data:
  userdata: I2Nsb3VkLWNvbmZpZwpwYXNzd29yZDogYXRvbWljCnNzaF9wd2F1dGg6IFRydWUKY2hwYXNzd2Q6IHsgZXhwaXJlOiBGYWxzZSB9Cg==
kind: Secret
metadata:
  annotations:


(cnv-tests) [kbidarka@localhost vm-secret]$ echo "I2Nsb3VkLWNvbmZpZwpwYXNzd29yZDogYXRvbWljCnNzaF9wd2F1dGg6IFRydWUKY2hwYXNzd2Q6IHsgZXhwaXJlOiBGYWxzZSB9Cg==" | base64 -d
#cloud-config
password: atomic
ssh_pwauth: True
chpasswd: { expire: False }


-------------------------------------------------------------------------

-------------------------------------------------------------------

spec:
  running: true
  template:
    metadata:
      creationTimestamp: null
      labels:
        kubevirt.io/domain: vm-nfs-rhel82
        kubevirt.io/size: small
    spec:
      domain:
        cpu:
          cores: 1
          sockets: 1
          threads: 1
        devices:
          disks:
          - disk:
              bus: virtio
            name: rootdisk
          - disk:
              bus: virtio
            name: cloudinitdisk
          interfaces:
          - masquerade: {}
            name: default
          networkInterfaceMultiqueue: true
          rng: {}
        machine:
          type: pc-q35-rhel8.2.0
        resources:
          requests:
            memory: 2Gi
      evictionStrategy: LiveMigrate
      networks:
      - name: default
        pod: {}
      terminationGracePeriodSeconds: 180
      volumes:
      - name: rootdisk
        persistentVolumeClaim:
          claimName: dv-nfs-rhel82
      - cloudInitConfigDrive:
          secretRef:
            name: my-vm-secret
        name: cloudinitdisk


--------------------------------------------------------------------

(cnv-tests) [kbidarka@localhost vm-secret]$ virtctl console vm-nfs-rhel82
Successfully connected to vm-nfs-rhel82 console. The escape sequence is ^]

[cloud-user@vm-nfs-rhel82 ~]$ exit
logout

Red Hat Enterprise Linux 8.2 (Ootpa)
Kernel 4.18.0-193.19.1.el8_2.x86_64 on an x86_64

Activate the web console with: systemctl enable --now cockpit.socket

vm-nfs-rhel82 login: cloud-user
Password: 
Last login: Mon Oct 19 15:21:09 on ttyS0
[cloud-user@vm-nfs-rhel82 ~]$ 


(cnv-tests) [kbidarka@localhost vm-secret]$ oc get pods
NAME                                READY     STATUS    RESTARTS   AGE
virt-launcher-vm-nfs-rhel82-jwvr2   1/1       Running   0          13m
(cnv-tests) [kbidarka@localhost vm-secret]$ oc rsh -c compute virt-launcher-vm-nfs-rhel82-jwvr2 
sh-4.4# cd /var/run/kubevirt-private/
sh-4.4# cd secret
sh-4.4# cd cloudinitdisk/
sh-4.4# ls
userdata
sh-4.4# cat userdata 
#cloud-config
password: atomic
ssh_pwauth: True
chpasswd: { expire: False }
sh-4.4# exit
exit
-----------------------------------------------------------------------


When using above, "userdata":
    Was able to login to the VMI using the passwd "atomic" successfully, which was set in the "userdata" base64 format as shown above.

I was under the impression that, the secret should work when using "userData". 

Not 100% sure, but need someone to confirm this.

-------------------

[kbidarka@localhost vm-secret]$ oc get secret my-vm-secret -o yaml 
apiVersion: v1
data:
  userData: I2Nsb3VkLWNvbmZpZwpwYXNzd29yZDogYXRvbWljCnNzaF9wd2F1dGg6IFRydWUKY2hwYXNzd2Q6IHsgZXhwaXJlOiBGYWxzZSB9Cg==
kind: Secret
metadata:

-------------------

Using the above secret, I see the below issue:

(cnv-tests) [kbidarka@localhost vm-secret]$ virtctl start vm-nfs-rhel82
VM vm-nfs-rhel82 was scheduled to start
(cnv-tests) [kbidarka@localhost vm-secret]$ oc get pods
NAME                                READY     STATUS    RESTARTS   AGE
virt-launcher-vm-nfs-rhel82-hx2h5   1/1       Running   0          18s
(cnv-tests) [kbidarka@localhost vm-secret]$ oc rsh virt-launcher-vm-nfs-rhel82-hx2h5
sh-4.4# cd /var/run/kubevirt-private/secret/cloudinitdisk/
sh-4.4# ls
userdata
sh-4.4# cd userdata/ 
sh-4.4# ls
sh-4.4# exit
exit
(cnv-tests) [kbidarka@localhost vm-secret]$ oc logs -f  virt-launcher-vm-nfs-rhel82-hx2h5

{"component":"virt-launcher","level":"error","msg":"internal error: Child process (/usr/sbin/dmidecode -q -t 0,1,2,3,4,17) unexpected exit status 1: /dev/mem: No such file or directory","pos":"virCommandWait:2713","subcomponent":"libvirt","thread":"49","timestamp":"2020-10-19T20:18:57.259000Z"}
{"component":"virt-launcher","level":"info","msg":"Connected to libvirt daemon","pos":"libvirt.go:396","timestamp":"2020-10-19T20:18:57.697496Z"}
{"component":"virt-launcher","level":"info","msg":"Registered libvirt event notify callback","pos":"client.go:394","timestamp":"2020-10-19T20:18:57.701405Z"}
{"component":"virt-launcher","level":"info","msg":"Marked as ready","pos":"virt-launcher.go:71","timestamp":"2020-10-19T20:18:57.701707Z"}
{"component":"virt-launcher","kind":"","level":"info","msg":"Executing PreStartHook on VMI pod environment","name":"vm-nfs-rhel82","namespace":"default","pos":"manager.go:982","timestamp":"2020-10-19T20:18:58.301361Z","uid":"d29a8b14-0b36-40c9-9767-1f05e66f1f56"}
{"component":"virt-launcher","level":"error","msg":"could not read secret data from source: /var/run/kubevirt-private/secret/cloudinitdisk/userdata","pos":"cloud-init.go:207","reason":"read /var/run/kubevirt-private/secret/cloudinitdisk/userdata: is a directory","timestamp":"2020-10-19T20:18:58.301548Z"}
{"component":"virt-launcher","level":"error","msg":"could not read secret data from source: /var/run/kubevirt-private/secret/cloudinitdisk/networkdata","pos":"cloud-init.go:207","reason":"open /var/run/kubevirt-private/secret/cloudinitdisk/networkdata: no such file or directory","timestamp":"2020-10-19T20:18:58.301639Z"}
{"component":"virt-launcher","level":"error","msg":"could not read secret data from source: /var/run/kubevirt-private/secret/cloudinitdisk/networkData","pos":"cloud-init.go:207","reason":"open /var/run/kubevirt-private/secret/cloudinitdisk/networkData: no such file or directory","timestamp":"2020-10-19T20:18:58.301712Z"}
{"component":"virt-launcher","kind":"","level":"error","msg":"pre start setup for VirtualMachineInstance failed.","name":"vm-nfs-rhel82","namespace":"default","pos":"manager.go:1258","reason":"no cloud-init data-source found at volume: cloudinitdisk","timestamp":"2020-10-19T20:18:58.301748Z","uid":"d29a8b14-0b36-40c9-9767-1f05e66f1f56"}
{"component":"virt-launcher","kind":"","level":"error","msg":"Failed to sync vmi","name":"vm-nfs-rhel82","namespace":"default","pos":"server.go:161","reason":"no cloud-init data-source found at volume: cloudinitdisk","timestamp":"2020-10-19T20:18:58.301853Z","uid":"d29a8b14-0b36-40c9-9767-1f05e66f1f56"}

Also from the above virt-launcher pod logs, for some reason it is not looking for the "/var/run/kubevirt-private/secret/cloudinitdisk/userData". Should we be expecting it to look for that dir as well?

(In reply to Kedar Bidarkar from comment #6)
> --------------------------------------------------------------------
> [kbidarka@localhost vm-secret]$ cat my-vm-secret.yaml 
> apiVersion: v1
> kind: Secret
> metadata:
>   name: my-vm-secret
> type: Opaque
> data:
>   userData:
> I2Nsb3VkLWNvbmZpZwpwYXNzd29yZDogYXRvbWljCnNzaF9wd2F1dGg6IFRydWUKY2hwYXNzd2Q6I
> HsgZXhwaXJlOiBGYWxzZSB9Cg==
> 
> [kbidarka@localhost vm-secret]$ oc apply -f my-vm-secret.yaml 
> [kbidarka@localhost vm-secret]$ oc get secret my-vm-secret -o yaml 
> apiVersion: v1
> data:
>   userdata:
> I2Nsb3VkLWNvbmZpZwpwYXNzd29yZDogYXRvbWljCnNzaF9wd2F1dGg6IFRydWUKY2hwYXNzd2Q6I
> HsgZXhwaXJlOiBGYWxzZSB9Cg==
> kind: Secret
> metadata:
>   annotations:
> 
> 

I'm confused, the file says "userData" but on creation it was changed to "userdata"?
Is this maybe a copy-paste error? I just tested this on k8s and the string was still "userData" after creation.

Also later on, the virt-launcher pod is shown as running fine, when its logs show a critical error.

Please be more explicit on what works and what doesn't.

Thank you!

Assumed yes to the above.

Using the yaml from the previous comments, I was able to reproduce and fix the issue.
See https://github.com/kubevirt/kubevirt/pull/4670

To verify: follow reproduction steps in description

[kbidarka@localhost secret-gcp-ipi]$ cat my-vm-secret.yaml 
apiVersion: v1
kind: Secret
metadata:
  name: my-vm-secret
type: Opaque
data:
  userData: I2Nsb3VkLWNvbmZpZwpwYXNzd29yZDogYXRvbWljCnNzaF9wd2F1dGg6IFRydWUKY2hwYXNzd2Q6IHsgZXhwaXJlOiBGYWxzZSB9Cg==

[kbidarka@localhost secret-gcp-ipi]$ cat secretref-configdrive.yaml | grep -B 5 -A 10 ConfigDrive
      terminationGracePeriodSeconds: 0
      volumes:
      - dataVolume:
          name: rhel84-secref-dv2
        name: datavolumedisk1
      - cloudInitConfigDrive:
          secretRef:
            name: my-vm-secret
        name: cloudinitdisk



[kbidarka@localhost secret-gcp-ipi]$ virtctl console vm2-rhel84-secref
Successfully connected to vm2-rhel84-secref console. The escape sequence is ^]

Red Hat Enterprise Linux 8.4 Beta (Ootpa)
Kernel 4.18.0-287.el8.dt4.x86_64 on an x86_64

Activate the web console with: systemctl enable --now cockpit.socket

vm2-rhel84-secref login: cloud-user
Password: 
[root@vm2-rhel84-secref ~]# (cnv-tests) [kbidarka@localhost secret-gcp-ipi]$ 
[kbidarka@localhost secret-gcp-ipi]$ oc get pods
NAME                                    READY   STATUS      RESTARTS   AGE
virt-launcher-vm2-rhel84-6c9dr          0/1     Completed   0          44h
virt-launcher-vm2-rhel84-m26dv          1/1     Running     0          51m
virt-launcher-vm2-rhel84-secref-sl94h   1/1     Running     0          3m12s

[kbidarka@localhost secret-gcp-ipi]$ oc rsh virt-launcher-vm2-rhel84-secref-sl94h 
sh-4.4# cd /var/run/kubevirt-private/secret/cloudinitdisk/

sh-4.4# ls
userData  userdata
sh-4.4# cat userData 
#cloud-config
password: atomic
ssh_pwauth: True
chpasswd: { expire: False }


Summary: Was able to successfully login to the VM with secret using "userData".

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.8.0 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2920