Description of problem: After some of the latest updates to Fedora (in the last week or so, I can't say exactly when), the Incus agent will no longer run in virtual machines. After opening a bug report with the upstream project, the author pointed me towards the fact that the incus-agent binary is dynamically linked on Fedora (https://github.com/lxc/incus/issues/2398). After retrying with a statically linked binary taken from their releases page, the issue was no longer present. Version-Release number of selected component (if applicable): 6.15 How reproducible: Always. Steps to Reproduce: 1.Create a VM: incus launch images:ubuntu/24.04 testvm --vm 2. Wait the few seconds it would typically take to launch the agent 3. Attempt to log in: incus exec testvm -- bash Actual results: An error saying "websocket: bad handshake" or "VM agent isn't currently running". Expected results: An interactive shell is spawned in the VM. Additional info: It seems that Fedora should not be dynamically linking the agent binary, as it will then fail to run unless the VM instance is exactly the same operating system + version as the host.
Greetings. You mentioned that this error occurs after a recent update. Do you know which version of incus that this was working?
It was last working on 6.14, and when I manually copy the agent binary from the 6.14-1 package to /usr/bin/incus-agent, it works again. The 6.14-1 agent binary is also dynamically linked.
Ok. I have not had a chance to recreate this yet... I will run some test scenarios and provide updates here. In the upstream ticket you mentioned that this happens on Fedora 42. I will focus my test efforts there.
I can add some detail... My host is Fedora 42 with... Installed packages incus.x86_64 6.15-1.fc42 updates incus-agent.x86_64 6.15-1.fc42 updates incus-client.x86_64 6.15-1.fc42 updates incus-selinux.noarch 6.15-1.fc42 updates incus-tools.x86_64 6.15-1.fc42 updates It appears that incus-agent is built with dynamic linking so when run in a VM with a recent glibc it works... In a Ubuntu 24.04.3 VM with libc6/noble-updates,noble-security,now 2.39-0ubuntu8.5 amd64 # ldd /run/incus_agent/incus-agent linux-vdso.so.1 (0x00007226a90a6000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007226a7c00000) /lib64/ld-linux-x86-64.so.2 (0x00007226a90a8000) But in older distros with older glibcs it doesn't... In a Ubuntu 22.04.5 VM with libc6/jammy-updates,jammy-security,now 2.35-0ubuntu3.10 amd64 # ldd /run/incus_agent/incus-agent /run/incus_agent/incus-agent: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_ABI_DT_RELR' not found (required by /run/incus_agent/incus-agent) linux-vdso.so.1 (0x00007ffe66dc7000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2c0bb05000) /lib64/ld-linux-x86-64.so.2 (0x00007f2c0cf47000) So, any binaries that will run inside a VM or container need to be statically linked.
Created attachment 2107119 [details] Link incus-agent statically This patch changes the incus.spec file to add '-ldflags="-extldflags -static"' to the incus-agent gobuild. I've verified that the resulting binary is indeed statically linked and runs with older glibcs.
Hey, I'm having the issue as well, I spent quite some time on this issue. I dont think its related to incus-agent being dyamically linked instead of static. I've tried patch from George Joseph, while it indeed link incus-agent statically it does not fix the issue and symptoms are the same. Still link agent statically should be done as of incus developer recommendation. I've tried using incus-agent 6.15 built by the developer (static, binary found in github release) -> same symptoms Using agent 6.14 built by developer (static) -> works as expected Using agent 6.17 built by developer (static) -> works as expected I think issue is likely related to something in incus 6.15 which I could not troubleshoot nor find a related issue on github I think updating the package to 6.16 or 6.17 should be done to fix this
PACordonnier, after you rebuilt 6.15 or copied from github, did you verify that in the VM, incus-agent is the one you think it should be? `$ file /run/incus_agent/incus-agent` /run/incus_agent/incus-agent: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=05bf548ce0b85bded1076a707a781ba9bfdda2f0, for GNU/Linux 3.2.0, not stripped, too many notes (256) Another issue might be where on the host incus-agent was installed. With the recent push to merge sbin and bin, things have gotten a bit confusing. If /usr/sbin isn't linked to /usr/bin, try putting incus-agent in both and see what happens.
It is absolutely the one I expect it to be. > On the guest `# file /var/run/incus_agent/incus-agent` /var/run/incus_agent/incus-agent: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=19974e3346d0d9b3be1ab51b86af67ae7362b8ae, with debug_info, not stripped > on the host `$ file /usr/bin/incus-agent` /usr/bin/incus-agent: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=19974e3346d0d9b3be1ab51b86af67ae7362b8ae, with debug_info, not stripped On my system /usr/sbin is linked to /usr/bin
The relevant section about statically built Incus agent binaries is in the "Packaging recommendations" document: https://linuxcontainers.org/incus/docs/main/packaging/#incus-agent-binaries It is worth mentioning, that if one is managing different VM (Linux, Windows, etc.) on multiple architectures (arm64, x64_64), it would be beneficial, if the incus package would ship with all the required incus agent binaries pre built.
FEDORA-2025-ba21fcf444 (incus-6.18-2.fc42) has been submitted as an update to Fedora 42. https://bodhi.fedoraproject.org/updates/FEDORA-2025-ba21fcf444
FEDORA-2025-0ad53ef7cd (incus-6.18-2.fc43) has been submitted as an update to Fedora 43. https://bodhi.fedoraproject.org/updates/FEDORA-2025-0ad53ef7cd
FEDORA-2025-0ad53ef7cd has been pushed to the Fedora 43 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-0ad53ef7cd` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-0ad53ef7cd See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2025-ba21fcf444 has been pushed to the Fedora 42 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-ba21fcf444` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-ba21fcf444 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2025-9cbe00a763 has been pushed to the Fedora 42 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-9cbe00a763` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-9cbe00a763 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2025-2c3612f5ce has been pushed to the Fedora 43 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-2c3612f5ce` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-2c3612f5ce See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2025-2c3612f5ce (incus-6.19.1-1.fc43) has been pushed to the Fedora 43 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2025-9cbe00a763 (incus-6.19.1-1.fc42) has been pushed to the Fedora 42 stable repository. If problem still persists, please make note of it in this bug report.