Description of problem: Fido containers can be found at https://quay.io/organization/fido-fdo, download these containers, configure and start them in podman, then boot edge vm to perform device onboarding, failed at serviceinfo step. This problem was found when run fdo container test script https://github.com/virt-s1/rhel-edge/blob/main/ostree-fdo-container.sh. This test script can pass without error before but failed yesterday. Is this error caused by fido key files? I can see this in error message "SSHkey module missing username or key", you can find all log in reproduce steps. Version-Release number of selected component (if applicable): 9.3 How reproducible: 100% Steps to Reproduce: 1.Generate fido key files. podman run -v "$PWD"/aio/:/aio:z \ "quay.io/fido-fdo/aio:nightly" \ aio --directory aio generate-configs-and-keys --contact-hostname "$FDO_MANUFACTURING_ADDRESS" 2.Use these key files in fido containers, and start them in podman. here is the example for manufacturing server. cp aio/keys/* fdo/keys/ sudo podman run -d \ --ip "$FDO_MANUFACTURING_ADDRESS" \ --name manufacture-server \ --network edge \ -v "$PWD"/fdo/:/etc/fdo/:z \ -p 8080:8080 \ "quay.io/fido-fdo/manufacturing-server:nightly" 3. Wait and check all fido containers are up and running. 4. Build edge-container images with blueprint like this: name = "container" description = "A base rhel-edge container image" version = "0.0.1" modules = [] groups = [] [[packages]] name = "python3" version = "*" 5. Start container in podman, and build edge-simplified-installer like this: name = "installer" description = "A rhel-edge simplified-installer image" version = "0.0.1" modules = [] groups = [] [customizations] installation_device = "/dev/vda" [customizations.fdo] manufacturing_server_url="http://${FDO_MANUFACTURING_ADDRESS}:8080" diun_pub_key_insecure="true" [[customizations.user]] name = "admin" description = "Administrator account" password = "***PASSWORD***" key = "${SSH_KEY_PUB}" home = "/home/admin/" groups = ["wheel"] 6. Provision vm with edge-simplified-installer image and boot it. 7. Check fido log in vm: Jun 19 04:36:06 vm-1 systemd[1]: Starting FDO client... Jun 19 04:36:06 vm-1 fdo-client-linuxapp[1191]: 2023-06-19T08:36:06.746Z INFO fdo_client_linuxapp > Found device credential at FileSystemPath { path: "/boot/device-credentials", deactivation_method: None } Jun 19 04:36:06 vm-1 fdo-client-linuxapp[1191]: 2023-06-19T08:36:06.877Z INFO fdo_client_linuxapp > Got TO2 addresses: ["http://192.168.200.51:8081"] Jun 19 04:36:06 vm-1 fdo-client-linuxapp[1191]: 2023-06-19T08:36:06.878Z INFO fdo_client_linuxapp > Performing TO2 protocol, URL: "http://192.168.200.51:8081" Jun 19 04:36:06 vm-1 fdo-client-linuxapp[1191]: 2023-06-19T08:36:06.986Z INFO fdo_client_linuxapp::serviceinfo > Creating file "/etc/sudoers.d/fdouser" with 32 bytes (mode 384) Jun 19 04:36:06 vm-1 fdo-client-linuxapp[1191]: 2023-06-19T08:36:06.989Z INFO fdo_client_linuxapp::serviceinfo > Initiating disk re-encryption, disk-label: /dev/vda4, pin: tpm2, config: {}, reencrypt: true Jun 19 04:36:08 vm-1 fdo-client-linuxapp[1191]: 2023-06-19T08:36:08.162Z ERROR fdo_client_linuxapp > ServiceInfo failed, error: Error processing returned serviceinfo Jun 19 04:36:08 vm-1 fdo-client-linuxapp[1191]: Caused by: Jun 19 04:36:08 vm-1 fdo-client-linuxapp[1191]: SSHkey module missing username or key Jun 19 04:36:08 vm-1 fdo-client-linuxapp[1191]: 2023-06-19T08:36:08.174Z ERROR fdo_client_linuxapp > Error performing TO2 ownership protocol Jun 19 04:36:08 vm-1 fdo-client-linuxapp[1191]: Caused by: Jun 19 04:36:08 vm-1 fdo-client-linuxapp[1191]: Error performing the ServiceInfo roundtrips with TO2 address http://192.168.200.51:8081 Jun 19 04:37:56 vm-1 fdo-client-linuxapp[1191]: 2023-06-19T08:37:56.428Z INFO fdo_client_linuxapp > Got TO2 addresses: ["http://192.168.200.51:8081"] Jun 19 04:37:56 vm-1 fdo-client-linuxapp[1191]: 2023-06-19T08:37:56.428Z INFO fdo_client_linuxapp > Performing TO2 protocol, URL: "http://192.168.200.51:8081" Jun 19 04:37:56 vm-1 fdo-client-linuxapp[1191]: 2023-06-19T08:37:56.501Z INFO fdo_client_linuxapp::serviceinfo > Creating file "/etc/sudoers.d/fdouser" with 32 bytes (mode 384) Jun 19 04:37:56 vm-1 fdo-client-linuxapp[1191]: 2023-06-19T08:37:56.506Z INFO fdo_client_linuxapp::serviceinfo > Initiating disk re-encryption, disk-label: /dev/vda4, pin: tpm2, config: {}, reencrypt: true Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]: 2023-06-19T08:37:57.404Z ERROR fdo_client_linuxapp > ServiceInfo failed, error: Error processing returned serviceinfo Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]: Caused by: Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]: 0: Error executing clevis Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]: 1: Error executing disk encryption for disk label /dev/vda4 Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]: 2: Error rebinding clevis Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]: 3: Error binding clevis Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]: 4: Failed to bind clevis: ExitStatus(unix_wait_status(256)), stdout: , stderr: Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]: Operation incompatible with device marked for LUKS2 reencryption. Aborting. Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]: Unable to save/update key slot; operation cancelled Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]: Error adding new binding to /dev/vda4 Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]: Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]: 2023-06-19T08:37:57.412Z ERROR fdo_client_linuxapp > Error performing TO2 ownership protocol Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]: Caused by: Jun 19 04:37:57 vm-1 fdo-client-linuxapp[1191]: Error performing the ServiceInfo roundtrips with TO2 address http://192.168.200.51:8081 Actual results: fdo onboarding failed. Expected results: should pass without error. Additional info:
We need to know how the serviceinfo-api-server config file looks like.
I use this config file https://github.com/virt-s1/rhel-edge/blob/main/data/fdo/serviceinfo-api-server.yml: --- service_info: initial_user: null files: null commands: null diskencryption_clevis: - disk_label: /dev/vda4 reencrypt: true binding: pin: tpm2 config: '{}' additional_serviceinfo: null bind: 0.0.0.0:8083 service_info_auth_token: 2IOtlXsSqfcGjnhBLZjPiHIteskzZEW3lncRzpEmgqI= admin_auth_token: Va40bSkLcxwnfml1pmIuaWaOZG96mSMB6fu0xuzcueg= device_specific_store_driver: Directory: path: /etc/fdo/stores/serviceinfo_api_devices
PR to fix the issue posted: https://github.com/fedora-iot/fido-device-onboard-rs/pull/519
Tested new fdo containers generated by https://github.com/fedora-iot/fido-device-onboard-rs/pull/519 Still see some error messages, but different from the previous error we were getting, at least the luks part is fixed. Jun 26 22:55:57 vm-1 systemd[1]: Starting FDO client... Jun 26 22:55:57 vm-1 fdo-client-linuxapp[1184]: 2023-06-27T02:55:57.751Z INFO fdo_client_linuxapp > Found device credential at FileSystemPath { path: "/boot/device-credentials", deactivation_method: None } Jun 26 22:55:57 vm-1 fdo-client-linuxapp[1184]: 2023-06-27T02:55:57.866Z ERROR fdo_client_linuxapp > Error performing TO1 with ServiceClient { protocol_version: Version1_1, base_url: "http://192.168.200.52:8082", client: Client { accepts: Accepts, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "/"} }, authorization_token: None, encryption_keys: EncryptionKeys { cipher_suite: None, keys: None }, last_message_type: Some(Error), non_interoperable_kdf_required: Some(false) } Jun 26 22:55:57 vm-1 fdo-client-linuxapp[1184]: 2023-06-27T02:55:57.867Z ERROR fdo_client_linuxapp > Error Couldn't get TO1 from any Rendezvous server! getting usable To1d from rv_entry RendezvousInterpretedDirective { ip_addresses: Some([IPAddress(192.168.200.52)]), dns_name: None, port: 8082, server_certificate_hash: None, ca_certificate_hash: None, user_input: false, wifi_ssid: None, wifi_password: None, medium: None, protocol: Http, delay: 0, bypass: false } Jun 26 22:57:46 vm-1 fdo-client-linuxapp[1184]: 2023-06-27T02:57:46.912Z INFO fdo_client_linuxapp > Got TO2 addresses: ["http://192.168.200.51:8081"] Jun 26 22:57:46 vm-1 fdo-client-linuxapp[1184]: 2023-06-27T02:57:46.912Z INFO fdo_client_linuxapp > Performing TO2 protocol, URL: "http://192.168.200.51:8081" Jun 26 22:57:47 vm-1 fdo-client-linuxapp[1184]: 2023-06-27T02:57:47.009Z INFO fdo_client_linuxapp::serviceinfo > Creating file "/etc/sudoers.d/fdouser" with 32 bytes (mode 384) Jun 26 22:57:47 vm-1 fdo-client-linuxapp[1184]: 2023-06-27T02:57:47.014Z INFO fdo_client_linuxapp::serviceinfo > Initiating disk re-encryption, disk-label: /dev/vda4, pin: tpm2, config: {}, reencrypt: true Jun 26 22:57:48 vm-1 fdo-client-linuxapp[1184]: 2023-06-27T02:57:48.204Z ERROR fdo_client_linuxapp > ServiceInfo failed, error: Error processing returned serviceinfo Jun 26 22:57:48 vm-1 fdo-client-linuxapp[1184]: Caused by: Jun 26 22:57:48 vm-1 fdo-client-linuxapp[1184]: SSHkey module missing username or key Jun 26 22:57:48 vm-1 fdo-client-linuxapp[1184]: 2023-06-27T02:57:48.211Z ERROR fdo_client_linuxapp > Error performing TO2 ownership protocol Jun 26 22:57:48 vm-1 fdo-client-linuxapp[1184]: Caused by: Jun 26 22:57:48 vm-1 fdo-client-linuxapp[1184]: Error performing the ServiceInfo roundtrips with TO2 address http://192.168.200.51:8081 Jun 26 22:59:23 vm-1 fdo-client-linuxapp[1184]: 2023-06-27T02:59:23.253Z INFO fdo_client_linuxapp > Got TO2 addresses: ["http://192.168.200.51:8081"] Jun 26 22:59:23 vm-1 fdo-client-linuxapp[1184]: 2023-06-27T02:59:23.254Z INFO fdo_client_linuxapp > Performing TO2 protocol, URL: "http://192.168.200.51:8081" Jun 26 22:59:23 vm-1 fdo-client-linuxapp[1184]: 2023-06-27T02:59:23.314Z INFO fdo_client_linuxapp::serviceinfo > Creating file "/etc/sudoers.d/fdouser" with 32 bytes (mode 384) Jun 26 22:59:23 vm-1 fdo-client-linuxapp[1184]: 2023-06-27T02:59:23.319Z INFO fdo_client_linuxapp::serviceinfo > Initiating disk re-encryption, disk-label: /dev/vda4, pin: tpm2, config: {}, reencrypt: true Jun 26 22:59:24 vm-1 fdo-client-linuxapp[1184]: 2023-06-27T02:59:24.291Z ERROR fdo_client_linuxapp > ServiceInfo failed, error: Error processing returned serviceinfo
Verified with new fido containers and new fdo client, worked as expected. Steps: 1. Download fido containers and start in podman. 2. Build simplified installer image with fdo settings in blueprint, boot it. 3. All check passed, no problem found.