Fedora Account System
Red Hat Associate
Red Hat Customer
The incus-agent binary provided by the Fedora package is dynamically linked, which causes "websocket: bad handshake" errors when using `incus exec` on virtual machines. The incus-agent is injected into VMs at boot time via a 9p/virtiofs mount and must be statically linked to work correctly regardless of the guest OS libraries. $ file /usr/bin/incus-agent /usr/bin/incus-agent: ELF 64-bit LSB pie executable, x86-64, dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2 Upstream provides static binaries that work correctly: https://github.com/lxc/incus/releases (bin.linux.incus-agent.x86_64) Reproducible: Always Steps to Reproduce: 1. Install incus and incus-agent packages 2. Create a VM: incus launch images:ubuntu/24.04 testvm --vm 3. Wait for the VM to boot 4. Execute a command: incus exec testvm id Actual Results: Error: websocket: bad handshake Expected Results: Command executes successfully inside the VM Additional Information: Workaround: Replace with static binary from upstream: curl -L -o /usr/bin/incus-agent https://github.com/lxc/incus/releases/latest/download/bin.linux.incus-agent.x86_64 References: - Debian fixed this in incus 6.0.5-2: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1114918 - Forum discussion: https://discuss.linuxcontainers.org/t/incus-exec-failing-for-vms-with-a-websocket-bad-handshake/24865
Same issue confirmed on Fedora 43: [root@fedora ~]# grep PRETTY /etc/os-release PRETTY_NAME="Fedora Linux 43" [root@fedora ~]# file /usr/bin/incus-agent /usr/bin/incus-agent: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=957004a18c107c93a8e7252d0c563c03d4131721, for GNU/Linux 3.2.0, stripped The binary is still dynamically linked, causing the same "websocket: bad handshake" error on VMs.
FEDORA-2025-77328eb306 (incus-6.18-2.fc44) has been submitted as an update to Fedora 44. https://bodhi.fedoraproject.org/updates/FEDORA-2025-77328eb306
FEDORA-2025-77328eb306 (incus-6.18-2.fc44) has been pushed to the Fedora 44 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2025-ba21fcf444 (incus-6.18-2.fc42) has been submitted as an update to Fedora 42. https://bodhi.fedoraproject.org/updates/FEDORA-2025-ba21fcf444
FEDORA-2025-0ad53ef7cd (incus-6.18-2.fc43) has been submitted as an update to Fedora 43. https://bodhi.fedoraproject.org/updates/FEDORA-2025-0ad53ef7cd
FEDORA-2025-0ad53ef7cd has been pushed to the Fedora 43 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-0ad53ef7cd` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-0ad53ef7cd See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2025-ba21fcf444 has been pushed to the Fedora 42 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-ba21fcf444` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-ba21fcf444 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2025-9cbe00a763 has been pushed to the Fedora 42 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-9cbe00a763` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-9cbe00a763 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2025-2c3612f5ce has been pushed to the Fedora 43 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-2c3612f5ce` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-2c3612f5ce See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Hello, The issue is not fully fixed, My debug attempt here : https://bodhi.fedoraproject.org/updates/FEDORA-2025-ba21fcf444#comment-4470993 Regards, Julien
FEDORA-2025-2c3612f5ce (incus-6.19.1-1.fc43) has been pushed to the Fedora 43 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2025-9cbe00a763 (incus-6.19.1-1.fc42) has been pushed to the Fedora 42 stable repository. If problem still persists, please make note of it in this bug report.
The problem still persist in 6.19
Can confirm the problem persists. ``` ~ ➜ incus version Client version: 6.19.1 Server version: 6.19.1 ``` ``` ~ ➜ incus exec debian-vm -- bash Error: websocket: bad handshake ```
The incus-agent binary built via the spec's custom %gobuild_static macro fails at runtime with "websocket: bad handshake" when communicating with the incus daemon inside VMs. A manually built incus-agent using the same Go version and source works correctly. Root cause: The %gobuild_static macro uses GO111MODULE=off (via %gomodulesmode), which puts Go in legacy GOPATH mode. In this mode, vendored dependency resolution relies on GOPATH directory structure and symlinks set up by %goprep, rather than on go.mod and vendor/modules.txt. Testing with 3 build variants confirms that the discriminating factor is GO111MODULE=off vs GO111MODULE=on -mod=vendor: | Build | GO111MODULE | -mod=vendor | Result | |--------------------------------|---------------|-------------|--------| | Original %gobuild_static | off (GOPATH) | no | BROKEN | | Same flags, module mode | on | yes | works | | Minimal flags, GOPATH mode | off | no | BROKEN | | Minimal flags, module mode | on | yes | works | The GOPATH-mode vendor resolution (through the _build/src/github.com/lxc/incus symlink created by %goprep) produces a binary where the websocket handshake fails. Module-mode resolution (-mod=vendor reading vendor/modules.txt) produces a working binary. Note: /usr/share/gocode in the mock chroot is empty (only skeleton directories, no .go files), so this is not a package shadowing issue. The problem is probably in how GOPATH-mode resolves the vendor directory through the %goprep symlink structure. Fix: The %gobuild_static custom macro in incus.spec should use GO111MODULE=on -mod=vendor instead of %{?gomodulesmode} (GO111MODULE=off). This is the correct mode for a project with go.mod and a vendor/ directory. The change only affects the custom static build macro; all other binaries continue to use the standard %gobuild macro. ❯ git diff incus.spec diff --git a/incus.spec b/incus.spec index 1d850c9..5ae4611 100644 --- a/incus.spec +++ b/incus.spec @@ -19,13 +19,14 @@ Version: 6.19.1 # Set build macro for static builds +# Uses GO111MODULE=on -mod=vendor instead of %%gomodulesmode (GO111MODULE=off) +# to ensure vendored dependencies are correctly resolved (rhbz#XXXXXX) %define gocompilerflags_static -compiler gc -%define gobuild_baseflags_static %{gocompilerflags_static} -tags="rpm_crashtraceback ${GO_BUILDTAGS-${BUILDTAGS-}}" -a -v +%define gobuild_baseflags_static %{gocompilerflags_static} -mod=vendor -tags="rpm_crashtraceback ${GO_BUILDTAGS-${BUILDTAGS-}}" -a -v %define gobuild_ldflags_static ${GO_LDFLAGS-${LDFLAGS-}} %{?currentgoldflags} -B 0x$(echo "%{name}-%{version}-%{release}-${SOURCE_DATE_EPOCH:-}" | sha1sum | cut -d ' ' -f1) -compressdwarf=false %define gobuildflags_static() %{expand:%{gobuild_baseflags_static} -ldflags "%{gobuild_ldflags_static}"} %define gobuild_static(-) %{expand: - %{?gobuilddir:GOPATH="%{gobuilddir}:${GOPATH:+${GOPATH}:}%{?gopath}"} %{?gomodulesmode} \\ - CGO_ENABLED=0 go build %{gobuildflags_static} %{?**}; Rebuild the rpm : ❯ mock -r fedora-43-x86_64 (...) ❯ sudo dnf reinstall mock-results-dir/incus-agent-6.19.1-1.fc43.x86_64.rpm ❯ incus restart test; sleep 20 ❯ incus exec test id uid=0(root) gid=0(root) groups=0(root) This may also indicate a broader issue with %gomodulesmode (GO111MODULE=off) when used with projects that have vendored dependencies and go.mod — the GOPATH-mode vendor resolution through %goprep symlinks may not be fully equivalent to module-mode -mod=vendor resolution. Julien
Incomplete git diff in the previous message : ❯ git diff incus.spec diff --git a/incus.spec b/incus.spec index 1d850c9..5ae4611 100644 --- a/incus.spec +++ b/incus.spec @@ -19,13 +19,14 @@ Version: 6.19.1 # Set build macro for static builds +# Uses GO111MODULE=on -mod=vendor instead of %%gomodulesmode (GO111MODULE=off) +# to ensure vendored dependencies are correctly resolved (rhbz#XXXXXX) %define gocompilerflags_static -compiler gc -%define gobuild_baseflags_static %{gocompilerflags_static} -tags="rpm_crashtraceback ${GO_BUILDTAGS-${BUILDTAGS-}}" -a -v +%define gobuild_baseflags_static %{gocompilerflags_static} -mod=vendor -tags="rpm_crashtraceback ${GO_BUILDTAGS-${BUILDTAGS-}}" -a -v %define gobuild_ldflags_static ${GO_LDFLAGS-${LDFLAGS-}} %{?currentgoldflags} -B 0x$(echo "%{name}-%{version}-%{release}-${SOURCE_DATE_EPOCH:-}" | sha1sum | cut -d ' ' -f1) -compressdwarf=false %define gobuildflags_static() %{expand:%{gobuild_baseflags_static} -ldflags "%{gobuild_ldflags_static}"} %define gobuild_static(-) %{expand: - %{?gobuilddir:GOPATH="%{gobuilddir}:${GOPATH:+${GOPATH}:}%{?gopath}"} %{?gomodulesmode} \\ - CGO_ENABLED=0 go build %{gobuildflags_static} %{?**}; + GO111MODULE=on CGO_ENABLED=0 go build %{gobuildflags_static} %{?**}; } @@ -360,8 +361,11 @@ for cmd in incus fuidshift incus-benchmark lxc-to-incus lxd-to-incus; do done # Build incus-migrate and incus-agent statically (cf. rhbz#2419661) -BUILDTAGS="netgo" %gobuild_static -o %{gobuilddir}/bin/incus-migrate %{goipath}/cmd/incus-migrate -BUILDTAGS="agent netgo" %gobuild_static -o %{gobuilddir}/bin/incus-agent %{goipath}/cmd/incus-agent +# Uses GO111MODULE=on -mod=vendor, so paths must be relative to module root +pushd %{currentgosourcedir} +BUILDTAGS="netgo" %gobuild_static -o %{gobuilddir}/bin/incus-migrate ./cmd/incus-migrate +BUILDTAGS="agent netgo" %gobuild_static -o %{gobuilddir}/bin/incus-agent ./cmd/incus-agent +popd # build shell completions mkdir %{gobuilddir}/completions
FEDORA-2026-5af38b7ecb (incus-6.23-2.fc43) has been submitted as an update to Fedora 43. https://bodhi.fedoraproject.org/updates/FEDORA-2026-5af38b7ecb
Finally solved in https://bodhi.fedoraproject.org/updates/FEDORA-2026-5af38b7ecb https://src.fedoraproject.org/rpms/incus/c/6a4b7ba178ffc17a6d48aef7197d14c3b2c18a87?branch=f43
FEDORA-2026-5af38b7ecb has been pushed to the Fedora 43 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2026-5af38b7ecb` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2026-5af38b7ecb See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2026-094b7621cf has been pushed to the Fedora 43 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2026-094b7621cf` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2026-094b7621cf See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2026-094b7621cf (incus-6.23-3.fc43) has been pushed to the Fedora 43 stable repository. If problem still persists, please make note of it in this bug report.