Bug 2216111

Summary: fdo containers failed at serviceinfo with error SSHkey module missing username or key
Product: Red Hat Enterprise Linux 9 Reporter: Yi He <yih>
Component: fdo-serviceinfo-api-server-containerAssignee: idiez
Status: CLOSED COMPLETED QA Contact: Xiaofeng Wang <xiaofwan>
Severity: medium Docs Contact: Eliane Ramos Pereira <elpereir>
Priority: unspecified    
Version: 9.3CC: amurdaca, idiez, miabbott, perobins
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-06-28 14:52:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Yi He 2023-06-20 07:14:10 UTC
Description of problem:
Fido containers can be found at https://quay.io/organization/fido-fdo, download these containers, configure and start them in podman, then boot edge vm to perform device onboarding, failed at serviceinfo step.
This problem was found when run fdo container test script https://github.com/virt-s1/rhel-edge/blob/main/ostree-fdo-container.sh. This test script can pass without error before but failed yesterday.
Is this error caused by fido key files? I can see this in error message "SSHkey module missing username or key", you can find all log in reproduce steps.

Version-Release number of selected component (if applicable):
9.3

How reproducible:
100%

Steps to Reproduce:
1.Generate fido key files.
podman run -v "$PWD"/aio/:/aio:z \
  "quay.io/fido-fdo/aio:nightly" \
  aio --directory aio generate-configs-and-keys --contact-hostname "$FDO_MANUFACTURING_ADDRESS"

2.Use these key files in fido containers, and start them in podman. here is the example for manufacturing server.
cp aio/keys/* fdo/keys/
sudo podman run -d \
  --ip "$FDO_MANUFACTURING_ADDRESS" \
  --name manufacture-server \
  --network edge \
  -v "$PWD"/fdo/:/etc/fdo/:z \
  -p 8080:8080 \
  "quay.io/fido-fdo/manufacturing-server:nightly"

3. Wait and check all fido containers are up and running.

4. Build edge-container images with blueprint like this:
name = "container"
description = "A base rhel-edge container image"
version = "0.0.1"
modules = []
groups = []

[[packages]]
name = "python3"
version = "*"

5. Start container in podman, and build edge-simplified-installer like this:
name = "installer"
description = "A rhel-edge simplified-installer image"
version = "0.0.1"
modules = []
groups = []

[customizations]
installation_device = "/dev/vda"

[customizations.fdo]
manufacturing_server_url="http://${FDO_MANUFACTURING_ADDRESS}:8080"
diun_pub_key_insecure="true"

[[customizations.user]]
name = "admin"
description = "Administrator account"
password = "***PASSWORD***"
key = "${SSH_KEY_PUB}"
home = "/home/admin/"
groups = ["wheel"]

6. Provision vm with edge-simplified-installer image and boot it.

7. Check fido log in vm:
Jun 19 04:36:06 vm-1 systemd[1]: Starting FDO client...
Jun 19 04:36:06 vm-1 fdo-client-linuxapp[1191]:  2023-06-19T08:36:06.746Z INFO  fdo_client_linuxapp > Found device credential at FileSystemPath { path: "/boot/device-credentials", deactivation_method: None }
Jun 19 04:36:06 vm-1 fdo-client-linuxapp[1191]:  2023-06-19T08:36:06.877Z INFO  fdo_client_linuxapp > Got TO2 addresses: ["http://192.168.200.51:8081"]
Jun 19 04:36:06 vm-1 fdo-client-linuxapp[1191]:  2023-06-19T08:36:06.878Z INFO  fdo_client_linuxapp > Performing TO2 protocol, URL: "http://192.168.200.51:8081"
Jun 19 04:36:06 vm-1 fdo-client-linuxapp[1191]:  2023-06-19T08:36:06.986Z INFO  fdo_client_linuxapp::serviceinfo > Creating file "/etc/sudoers.d/fdouser" with 32 bytes (mode 384)
Jun 19 04:36:06 vm-1 fdo-client-linuxapp[1191]:  2023-06-19T08:36:06.989Z INFO  fdo_client_linuxapp::serviceinfo > Initiating disk re-encryption, disk-label: /dev/vda4, pin: tpm2, config: {}, reencrypt: true
Jun 19 04:36:08 vm-1 fdo-client-linuxapp[1191]:  2023-06-19T08:36:08.162Z ERROR fdo_client_linuxapp              > ServiceInfo failed, error: Error processing returned serviceinfo
Jun 19 04:36:08 vm-1 fdo-client-linuxapp[1191]: Caused by:
Jun 19 04:36:08 vm-1 fdo-client-linuxapp[1191]:     SSHkey module missing username or key
Jun 19 04:36:08 vm-1 fdo-client-linuxapp[1191]:  2023-06-19T08:36:08.174Z ERROR fdo_client_linuxapp              > Error performing TO2 ownership protocol
Jun 19 04:36:08 vm-1 fdo-client-linuxapp[1191]: Caused by:
Jun 19 04:36:08 vm-1 fdo-client-linuxapp[1191]:     Error performing the ServiceInfo roundtrips with TO2 address http://192.168.200.51:8081
Jun 19 04:37:56 vm-1 fdo-client-linuxapp[1191]:  2023-06-19T08:37:56.428Z INFO  fdo_client_linuxapp              > Got TO2 addresses: ["http://192.168.200.51:8081"]
Jun 19 04:37:56 vm-1 fdo-client-linuxapp[1191]:  2023-06-19T08:37:56.428Z INFO  fdo_client_linuxapp              > Performing TO2 protocol, URL: "http://192.168.200.51:8081"
Jun 19 04:37:56 vm-1 fdo-client-linuxapp[1191]:  2023-06-19T08:37:56.501Z INFO  fdo_client_linuxapp::serviceinfo > Creating file "/etc/sudoers.d/fdouser" with 32 bytes (mode 384)
Jun 19 04:37:56 vm-1 fdo-client-linuxapp[1191]:  2023-06-19T08:37:56.506Z INFO  fdo_client_linuxapp::serviceinfo > Initiating disk re-encryption, disk-label: /dev/vda4, pin: tpm2, config: {}, reencrypt: true
Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]:  2023-06-19T08:37:57.404Z ERROR fdo_client_linuxapp              > ServiceInfo failed, error: Error processing returned serviceinfo
Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]: Caused by:
Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]:     0: Error executing clevis
Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]:     1: Error executing disk encryption for disk label /dev/vda4
Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]:     2: Error rebinding clevis
Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]:     3: Error binding clevis
Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]:     4: Failed to bind clevis: ExitStatus(unix_wait_status(256)), stdout: , stderr:
Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]:        Operation incompatible with device marked for LUKS2 reencryption. Aborting.
Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]:        Unable to save/update key slot; operation cancelled
Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]:        Error adding new binding to /dev/vda4
Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]:
Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]:  2023-06-19T08:37:57.412Z ERROR fdo_client_linuxapp              > Error performing TO2 ownership protocol
Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]: Caused by:
Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]:     Error performing the ServiceInfo roundtrips with TO2 address http://192.168.200.51:8081


Actual results:
fdo onboarding failed.

Expected results:
should pass without error.


Additional info:

Comment 1 idiez 2023-06-21 09:31:39 UTC
We need to know how the serviceinfo-api-server config file looks like.

Comment 2 Yi He 2023-06-21 11:05:10 UTC
I use this config file https://github.com/virt-s1/rhel-edge/blob/main/data/fdo/serviceinfo-api-server.yml:

---
service_info:
  initial_user: null
  files: null
  commands: null
  diskencryption_clevis:
    - disk_label: /dev/vda4
      reencrypt: true
      binding:
        pin: tpm2
        config: '{}'
  additional_serviceinfo: null
bind: 0.0.0.0:8083
service_info_auth_token: 2IOtlXsSqfcGjnhBLZjPiHIteskzZEW3lncRzpEmgqI=
admin_auth_token: Va40bSkLcxwnfml1pmIuaWaOZG96mSMB6fu0xuzcueg=
device_specific_store_driver:
  Directory:
    path: /etc/fdo/stores/serviceinfo_api_devices

Comment 3 idiez 2023-06-22 15:41:14 UTC
PR to fix the issue posted: https://github.com/fedora-iot/fido-device-onboard-rs/pull/519

Comment 4 Yi He 2023-06-27 10:03:27 UTC
Tested new fdo containers generated by https://github.com/fedora-iot/fido-device-onboard-rs/pull/519

Still see some error messages, but different from the previous error we were getting, at least the luks part is fixed.

Jun 26 22:55:57 vm-1 systemd[1]: Starting FDO client...
Jun 26 22:55:57 vm-1 fdo-client-linuxapp[1184]:  2023-06-27T02:55:57.751Z INFO  fdo_client_linuxapp > Found device credential at FileSystemPath { path: "/boot/device-credentials", deactivation_method: None }
Jun 26 22:55:57 vm-1 fdo-client-linuxapp[1184]:  2023-06-27T02:55:57.866Z ERROR fdo_client_linuxapp > Error performing TO1 with ServiceClient { protocol_version: Version1_1, base_url: "http://192.168.200.52:8082", client: Client { accepts: Accepts, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "/"} }, authorization_token: None, encryption_keys: EncryptionKeys { cipher_suite: None, keys: None }, last_message_type: Some(Error), non_interoperable_kdf_required: Some(false) }
Jun 26 22:55:57 vm-1 fdo-client-linuxapp[1184]:  2023-06-27T02:55:57.867Z ERROR fdo_client_linuxapp > Error Couldn't get TO1 from any Rendezvous server! getting usable To1d from rv_entry RendezvousInterpretedDirective { ip_addresses: Some([IPAddress(192.168.200.52)]), dns_name: None, port: 8082, server_certificate_hash: None, ca_certificate_hash: None, user_input: false, wifi_ssid: None, wifi_password: None, medium: None, protocol: Http, delay: 0, bypass: false }
Jun 26 22:57:46 vm-1 fdo-client-linuxapp[1184]:  2023-06-27T02:57:46.912Z INFO  fdo_client_linuxapp > Got TO2 addresses: ["http://192.168.200.51:8081"]
Jun 26 22:57:46 vm-1 fdo-client-linuxapp[1184]:  2023-06-27T02:57:46.912Z INFO  fdo_client_linuxapp > Performing TO2 protocol, URL: "http://192.168.200.51:8081"
Jun 26 22:57:47 vm-1 fdo-client-linuxapp[1184]:  2023-06-27T02:57:47.009Z INFO  fdo_client_linuxapp::serviceinfo > Creating file "/etc/sudoers.d/fdouser" with 32 bytes (mode 384)
Jun 26 22:57:47 vm-1 fdo-client-linuxapp[1184]:  2023-06-27T02:57:47.014Z INFO  fdo_client_linuxapp::serviceinfo > Initiating disk re-encryption, disk-label: /dev/vda4, pin: tpm2, config: {}, reencrypt: true
Jun 26 22:57:48 vm-1 fdo-client-linuxapp[1184]:  2023-06-27T02:57:48.204Z ERROR fdo_client_linuxapp              > ServiceInfo failed, error: Error processing returned serviceinfo
Jun 26 22:57:48 vm-1 fdo-client-linuxapp[1184]: Caused by:
Jun 26 22:57:48 vm-1 fdo-client-linuxapp[1184]:     SSHkey module missing username or key
Jun 26 22:57:48 vm-1 fdo-client-linuxapp[1184]:  2023-06-27T02:57:48.211Z ERROR fdo_client_linuxapp              > Error performing TO2 ownership protocol
Jun 26 22:57:48 vm-1 fdo-client-linuxapp[1184]: Caused by:
Jun 26 22:57:48 vm-1 fdo-client-linuxapp[1184]:     Error performing the ServiceInfo roundtrips with TO2 address http://192.168.200.51:8081
Jun 26 22:59:23 vm-1 fdo-client-linuxapp[1184]:  2023-06-27T02:59:23.253Z INFO  fdo_client_linuxapp              > Got TO2 addresses: ["http://192.168.200.51:8081"]
Jun 26 22:59:23 vm-1 fdo-client-linuxapp[1184]:  2023-06-27T02:59:23.254Z INFO  fdo_client_linuxapp              > Performing TO2 protocol, URL: "http://192.168.200.51:8081"
Jun 26 22:59:23 vm-1 fdo-client-linuxapp[1184]:  2023-06-27T02:59:23.314Z INFO  fdo_client_linuxapp::serviceinfo > Creating file "/etc/sudoers.d/fdouser" with 32 bytes (mode 384)
Jun 26 22:59:23 vm-1 fdo-client-linuxapp[1184]:  2023-06-27T02:59:23.319Z INFO  fdo_client_linuxapp::serviceinfo > Initiating disk re-encryption, disk-label: /dev/vda4, pin: tpm2, config: {}, reencrypt: true
Jun 26 22:59:24 vm-1 fdo-client-linuxapp[1184]:  2023-06-27T02:59:24.291Z ERROR fdo_client_linuxapp              > ServiceInfo failed, error: Error processing returned serviceinfo

Comment 5 Yi He 2023-06-28 13:45:14 UTC
Verified with new fido containers and new fdo client, worked as expected.

Steps:
1. Download fido containers and start in podman.
2. Build simplified installer image with fdo settings in blueprint, boot it.
3. All check passed, no problem found.