Bug 2389081

Greetings.

You mentioned that this error occurs after a recent update.  Do you know which version of incus that this was working?

It was last working on 6.14, and when I manually copy the agent binary from the 6.14-1 package to /usr/bin/incus-agent, it works again. The 6.14-1 agent binary is also dynamically linked.

Ok.  I have not had a chance to recreate this yet...  I will run some test scenarios and provide updates here. In the upstream ticket you mentioned that this happens on Fedora 42.  I will focus my test efforts there.

I can add some detail...

My host is Fedora 42 with...
Installed packages
incus.x86_64                  6.15-1.fc42         updates
incus-agent.x86_64            6.15-1.fc42         updates
incus-client.x86_64           6.15-1.fc42         updates
incus-selinux.noarch          6.15-1.fc42         updates
incus-tools.x86_64            6.15-1.fc42         updates

It appears that incus-agent is built with dynamic linking so when run in a VM with a recent glibc it works...

In a Ubuntu 24.04.3 VM with libc6/noble-updates,noble-security,now 2.39-0ubuntu8.5 amd64
# ldd /run/incus_agent/incus-agent
	linux-vdso.so.1 (0x00007226a90a6000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007226a7c00000)
	/lib64/ld-linux-x86-64.so.2 (0x00007226a90a8000)

But in older distros with older glibcs it doesn't...

In a Ubuntu 22.04.5 VM with libc6/jammy-updates,jammy-security,now 2.35-0ubuntu3.10 amd64 
# ldd /run/incus_agent/incus-agent
/run/incus_agent/incus-agent: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_ABI_DT_RELR' not found (required by /run/incus_agent/incus-agent)
        linux-vdso.so.1 (0x00007ffe66dc7000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2c0bb05000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f2c0cf47000)

So, any binaries that will run inside a VM or container need to be statically linked.

Created attachment 2107119 [details]
Link incus-agent statically

This patch changes the incus.spec file to add '-ldflags="-extldflags -static"' to the incus-agent gobuild.  I've verified that the resulting binary is indeed statically linked and runs with older glibcs.

Hey,

I'm having the issue as well, I spent quite some time on this issue.

I dont think its related to incus-agent being dyamically linked instead of static.
I've tried patch from George Joseph, while it indeed link incus-agent statically it does not fix the issue and symptoms are the same. Still link agent statically should be done as of incus developer recommendation.
I've tried using incus-agent 6.15 built by the developer (static, binary found in github release) -> same symptoms
Using agent 6.14 built by developer (static) -> works as expected
Using agent 6.17 built by developer (static) -> works as expected

I think issue is likely related to something in incus 6.15 which I could not troubleshoot nor find a related issue on github
I think updating the package to 6.16 or 6.17 should be done to fix this

PACordonnier, after you rebuilt 6.15 or copied from github, did you verify that in the VM, incus-agent is the one you think it should be?  

`$ file /run/incus_agent/incus-agent`
/run/incus_agent/incus-agent: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=05bf548ce0b85bded1076a707a781ba9bfdda2f0, for GNU/Linux 3.2.0, not stripped, too many notes (256)

Another issue might be where on the host incus-agent was installed.  With the recent push to merge sbin and bin, things have gotten a bit confusing.  If /usr/sbin isn't linked to /usr/bin, try putting incus-agent in both and see what happens.

It is absolutely the one I expect it to be.

> On the guest
`# file /var/run/incus_agent/incus-agent`
/var/run/incus_agent/incus-agent: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=19974e3346d0d9b3be1ab51b86af67ae7362b8ae, with debug_info, not stripped

> on the host
`$ file /usr/bin/incus-agent`
/usr/bin/incus-agent: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=19974e3346d0d9b3be1ab51b86af67ae7362b8ae, with debug_info, not stripped

On my system /usr/sbin is linked to /usr/bin

The relevant section about statically built Incus agent binaries is in the "Packaging recommendations" document: https://linuxcontainers.org/incus/docs/main/packaging/#incus-agent-binaries

It is worth mentioning, that if one is managing different VM (Linux, Windows, etc.) on multiple architectures (arm64, x64_64), it would be beneficial, if the incus package would ship with all the required incus agent binaries pre built.

FEDORA-2025-ba21fcf444 (incus-6.18-2.fc42) has been submitted as an update to Fedora 42.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-ba21fcf444

FEDORA-2025-0ad53ef7cd (incus-6.18-2.fc43) has been submitted as an update to Fedora 43.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-0ad53ef7cd

FEDORA-2025-0ad53ef7cd has been pushed to the Fedora 43 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-0ad53ef7cd`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-0ad53ef7cd

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2025-ba21fcf444 has been pushed to the Fedora 42 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-ba21fcf444`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-ba21fcf444

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2025-ba21fcf444 has been pushed to the Fedora 42 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-ba21fcf444`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-ba21fcf444

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2025-0ad53ef7cd has been pushed to the Fedora 43 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-0ad53ef7cd`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-0ad53ef7cd

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2025-9cbe00a763 has been pushed to the Fedora 42 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-9cbe00a763`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-9cbe00a763

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2025-2c3612f5ce has been pushed to the Fedora 43 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-2c3612f5ce`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-2c3612f5ce

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2025-2c3612f5ce (incus-6.19.1-1.fc43) has been pushed to the Fedora 43 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2025-9cbe00a763 (incus-6.19.1-1.fc42) has been pushed to the Fedora 42 stable repository.
If problem still persists, please make note of it in this bug report.