Bug 2343301 - Can't run Waydroid anymore on non-SSE4.2 CPU
Summary: Can't run Waydroid anymore on non-SSE4.2 CPU
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Fedora
Classification: Fedora
Component: waydroid
Version: 41
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Alessandro Astone
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2025-02-01 00:08 UTC by Andre Robatino
Modified: 2025-03-09 22:17 UTC (History)
2 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2025-02-13 17:35:40 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
/var/lib/waydroid/waydroid.log (8.34 KB, text/plain)
2025-02-10 22:07 UTC, Andre Robatino
no flags Details
dmesg output immediately after "waydroid session start" (3.32 KB, text/plain)
2025-02-10 22:25 UTC, Andre Robatino
no flags Details

Description Andre Robatino 2025-02-01 00:08:57 UTC
I had Waydroid installed and running using the instructions at https://docs.waydro.id/usage/install-on-desktops#install-instructions . After being notified of a System and Vendor update, I applied them and restarted Waydroid. Immediately after that, Waydroid wouldn't start again and I continually get messages such as these in dmesg:

 1427.972942] binder: 9077:8377 cannot find target node
[ 1427.972949] binder: 8377:9077 transaction call to 0:0 failed 409/29189/-22, size 0-0 line 3151
[ 1428.955013] binder: 7240:7240 cannot find target node
[ 1428.955023] binder: 7240:7240 transaction call to 0:0 failed 410/29189/-22, size 0-0 line 3151

I have uninstalled following the instructions in the same URL and cleanly reinstalled. After doing this I immediately get the same errors and Waydroid won't run. I'm at a total loss. This could of course be an error in the System Image and have nothing to do with the waydroid package. Unfortunately I wasn't able to determine the proper System and Vendor URLs for installing an earlier image on the command line (like "waydroid init -c SYSTEM_CHANNEL -v VENDOR_CHANNEL -s GAPPS", but I don't know what proper values for SYSTEM_CHANNEL and VENDOR_CHANNEL look like). The instructions I've found only say how to do it graphically using the latest images. I'm surprised that if the latest images are broken, no one has replaced them yet.

Reproducible: Always

Comment 1 Andre Robatino 2025-02-01 01:14:07 UTC
The older images are at https://sourceforge.net/projects/waydroid/files/images/ (I noticed that when doing the init graphically, the URL points to sourceforge.net and ends in "/download"). I determined the download URLs for the files I wanted and tried

root@lenovo-pc:~# waydroid init -c https://pilotfiber.dl.sourceforge.net/project/waydroid/images/system/lineage/waydroid_x86_64/lineage-18.1-20241228-GAPPS-waydroid_x86_64-system.zip -v https://phoenixnap.dl.sourceforge.net/project/waydroid/images/vendor/waydroid_x86_64/lineage-18.1-20241228-MAINLINE-waydroid_x86_64-vendor.zip -s GAPPS
[20:09:12] x86_64 CPU does not support SSE4.2, falling back to x86...
[20:09:12] ERROR: Failed to get system OTA channel: https://pilotfiber.dl.sourceforge.net/project/waydroid/images/system/lineage/waydroid_x86_64/lineage-18.1-20241228-GAPPS-waydroid_x86_64-system.zip/lineage/waydroid_x86/GAPPS.json, error: 404
[20:09:12] See also: <https://github.com/waydroid>
Run 'waydroid log' for details.
root@lenovo-pc:~#

so I don't know the proper syntax for grabbing earlier images or even if it's possible, only that the default images don't work.

Comment 2 Andre Robatino 2025-02-01 02:19:17 UTC
I used the instructions at https://docs.waydro.id/faq/using-custom-waydroid-images to install system and vendor images from lineage-18.1-20241228-GAPPS-waydroid_x86_64-system.zip and lineage-18.1-20241228-MAINLINE-waydroid_x86_64-vendor.zip in /etc/waydroid-extra/images and did "sudo waydroid init -f" after uninstalling completely. I believe those were the working versions I had before. However, I get exactly the same dmesg error and again waydroid fails to start.

Comment 3 Andre Robatino 2025-02-01 16:30:04 UTC
Realized that my x86_64 CPU does not support SSE4.2 so I needed to use the x86 images. But even after installing those (the latest 20250201 x86 images) in /etc/waydroid-extra/images and doing the complete uninstall/reinstall I get the exact same dmesg errors.

Comment 4 Alessandro Astone 2025-02-03 18:23:51 UTC
Presumably this is a regression from 1.1.42-1?

I cannot reproduce. Perhaps the lack of SSE4.2 is relevant here?

Comment 5 Andre Robatino 2025-02-03 20:24:07 UTC
It WAS working immediately (I had run an app) just before I was prompted to upgrade the System and Vendor images. After clicking on the button to restart Waydroid, it never restarted and that's when the dmesg messages started appearing. I can stop them by running "waydroid session stop" but that's all I can do. So there were no Fedora updates between when it was working and when it wasn't. I've tried a complete uninstall and reinstall according to the instructions in https://docs.waydro.id/usage/install-on-desktops#install-instructions but it still won't run and the exact same messages appear in dmesg, including the "29189/-22, size 0-0 line 3151" part. I would think I should be able to google something for "waydroid 'cannot find target node'" but nothing I found helped. I don't even know what file "line 3151" is referring to.

Comment 6 Alessandro Astone 2025-02-03 20:26:34 UTC
(In reply to Alessandro Astone from comment #4)
> Presumably this is a regression from 1.1.42-1?

BTW this was meant to say "libgbinder 1.1.42-1"

Comment 7 Andre Robatino 2025-02-03 20:43:07 UTC
(In reply to Alessandro Astone from comment #6)
> (In reply to Alessandro Astone from comment #4)
> > Presumably this is a regression from 1.1.42-1?
> 
> BTW this was meant to say "libgbinder 1.1.42-1"

OK, I currently have libgbinder-1.1.42-1.fc41.x86_64 which I installed on Jan. 30, just after it was pushed to stable. The trouble I'm experiencing started earlier, on Jan. 28, that's when I got the System and Vendor updates and Waydroid wouldn't start again.

Comment 8 Andre Robatino 2025-02-10 05:41:41 UTC
I have a laptop with a CPU that supports SSE4.2, and was able to install and run Waydroid on that. Curiously, when I shut down or reboot, there are similar messages including "29189/-22, size 0-0 line 3151", even though Waydroid is working. Try running "journalctl -b -1 | grep 3151" on a machine with working Waydroid to see if the messages happened on the last boot. The difference is that on my machine where Waydroid doesn't work, I see those messages immediately after starting a Waydroid session and they only end when I stop the session.

Comment 9 Andre Robatino 2025-02-10 05:42:46 UTC
Oh, and both machines are fully updated (including libgbinder).

Comment 10 Alessandro Astone 2025-02-10 20:28:59 UTC
You will need to get the android journal with `sudo waydroid shell -- logcat -d | tee logcat.txt` while waydroid is running
and share logcat.txt

Preferably, after returning to the default images: `sudo waydroid init --force -c https://ota.waydro.id/system -v https://ota.waydroid.id/vendor -s GAPPS`

Comment 11 Andre Robatino 2025-02-10 21:45:34 UTC
The following is for the machine where Waydroid won't run graphically. I removed /etc/waydroid-extra and its contents and then did a complete uninstall and reinstall according to https://docs.waydro.id/usage/install-on-desktops#install-instructions . After this, the errors in dmesg started immediately. I was not able to get output for logcat.txt, which was 0 bytes:

andre@lenovo-pc:/home/tmp$ sudo waydroid shell -- logcat -d | tee logcat.txt
[16:42:11] WayDroid container is STOPPED
andre@lenovo-pc:/home/tmp$ sudo waydroid container start
ERROR: WayDroid container service is already running
andre@lenovo-pc:/home/tmp$ sudo waydroid shell -- logcat -d | tee logcat.txt
[16:42:28] WayDroid container is STOPPED

I only knew that the Waydroid session was running because the command "waydroid session stop" causes the errors to stop.

Comment 12 Alessandro Astone 2025-02-10 21:51:38 UTC
1. Stop the session
   waydroid session stop

2. Start the session
   waydroid session start

3. In another terminal, very quickly after starting the session:
   sudo waydroid logcat | tee logcat.txt
   This command will only terminate if the container quits on its own; otherwise, CTRL+C to interrupt the log flow.

`/var/lib/waydroid/waydroid.log` and the output in the terminal from step.2 might also be useful.

Comment 13 Andre Robatino 2025-02-10 22:06:15 UTC
I get no output after "waydroid session start", it just hangs (no message and I never get the prompt back, except with Ctrl-C). And the other command continues to return "WayDroid container is STOPPED". In the meantime dmesg is being spammed and I can only stop the messages with "waydroid session stop". Will attach /var/lib/waydroid/waydroid.log .

Comment 14 Andre Robatino 2025-02-10 22:07:14 UTC
Created attachment 2075943 [details]
/var/lib/waydroid/waydroid.log

Comment 15 Alessandro Astone 2025-02-10 22:16:33 UTC
Run `waydroid session stop` again;

In the next few seconds after `waydroid session start`, does `dmesg` show anything more interesting than the 4 logs you mentioned?

Comment 16 Andre Robatino 2025-02-10 22:25:51 UTC
Created attachment 2075944 [details]
dmesg output immediately after "waydroid session start"

This is the additional dmesg output for a couple seconds immediately after running "waydroid session start".

Comment 17 Alessandro Astone 2025-02-10 22:29:04 UTC
> [ 3877.199616] traps: init[27923] trap invalid opcode ip:f7033c1d sp:ffcac650 error:0 in libc.so[4bc1d,f7029000+ae000]

Looks like some SSE4.2 has sneaked in even in the x86 (non-64) build

Comment 18 Andre Robatino 2025-02-11 00:55:21 UTC
Looking back at my comments, I never tried the x86 version of the 20241228 System and Vendor images (the last I thought I had before being prompted to upgrade to 20250125). Downloading and installing those custom images, it works again. So 20241228 is good and 20250125 is bad.

Comment 19 Jeffrey Walton 2025-02-11 03:24:13 UTC
Android requires SSE4.2 + POPCNT for x86_64; and SSSE3 for x86. See <https://developer.android.com/ndk/guides/abis.html>.

I would expect Waydroid to have the same cpu feature requirements as Android itself. I would be surprised to learn Waydroid emulates SSE4.2 on x86_64.

Comment 20 Andre Robatino 2025-02-11 03:46:57 UTC
When I set up Waydroid, it explicitly recognizes that my x86_64 CPU doesn't support SSE4.2 and downloads the x86 images (not x86_64). The x86 images aren't supposed to require SSE4.2.

Comment 21 Andre Robatino 2025-02-12 04:08:15 UTC
Works fine with the 20250118 System and Vendor images (lineage-18.1-20250118-GAPPS-waydroid_x86-system.zip and lineage-18.1-20250118-MAINLINE-waydroid_x86-vendor.zip). Updating the System image to 20250125 triggers the problem so lineage-18.1-20250125-GAPPS-waydroid_x86-system.zip is the first affected image. The 20250125 Vendor image does not appear to be affected.

Comment 22 Andre Robatino 2025-02-12 18:34:22 UTC
Just discovered https://github.com/waydroid/waydroid/issues . There is no bug filed for this issue. Should I file one, or is it already known upstream?

Comment 23 Alessandro Astone 2025-02-12 18:41:43 UTC
I am the upstream maintainer; we can remain here :)

I would like to send you some android builds to test.
Is it ok if I ask you to delete your android data, in order to run my test builds?

Comment 24 Jeffrey Walton 2025-02-12 19:08:46 UTC
(In reply to Alessandro Astone from comment #23)
> I would like to send you some android builds to test.
> Is it ok if I ask you to delete your android data, in order to run my test
> builds?

Alessandro, if it helps... I help maintain some libraries that need ISA awareness. Here is how we support Android via build flags (like CFLAGS and CXXFLAGS):

    i686: <https://github.com/weidai11/cryptopp/blob/master/TestScripts/setenv-android.sh#L347>
    x86_64: <https://github.com/weidai11/cryptopp/blob/master/TestScripts/setenv-android.sh#L400>

The flags, like `-mssse3 -mfpmath=sse` and `-fno-addrsig -fno-experimental-isel` for i686 (your x86) was taken from the default build flags used by Android NDK.

Comment 25 Andre Robatino 2025-02-12 19:28:08 UTC
(In reply to Alessandro Astone from comment #23)
> I am the upstream maintainer; we can remain here :)
> 
> I would like to send you some android builds to test.
> Is it ok if I ask you to delete your android data, in order to run my test
> builds?

Yes, that's fine. Although when I was testing the different weeklies to see which one broke, I only needed to delete /var/lib/waydroid, not all the dirs listed in the command "sudo rm -rf /var/lib/waydroid /home/.waydroid ~/waydroid ~/.share/waydroid ~/.local/share/applications/*aydroid* ~/.local/share/waydroid" from https://docs.waydro.id/usage/install-on-desktops#install-instructions . Will I need to delete all of those including those in my home dir?

Comment 26 Andre Robatino 2025-02-12 19:36:20 UTC
Should also mention that I see from https://developer.android.com/ndk/guides/abis.html that the x86 images aren't supposed to require SSE4 at all. My CPU (Intel Core 2 Duo E8400 x2) supports SSE4.1 but not SSE4.2, so if necessary check for the presence of either. From my /proc/cpuinfo:

flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm pti tpr_shadow flexpriority vpid dtherm vnmi

Comment 27 Alessandro Astone 2025-02-12 21:19:40 UTC
> I see from https://developer.android.com/ndk/guides/abis.html that the x86 images aren't supposed to require SSE4 at all.

Unfortunately that is wrong. AOSP by default compiles libc with SSE4.1 instructions (but we patched those out). It should be correct instead that SSE4.2 is not used.

Looking for possibly relevant changes around the dates you've identified, we've switched the malloc implementation on x86* from jemalloc-svelte to scudo, which might not be correctly configured for non-SSE4.2 targets.
Here's a build with that change reverted: https://sourceforge.net/projects/aleasto-lineageos/files/waydroid-x86.tar.gz/download

You can try without deleting any data for now.

Comment 28 Andre Robatino 2025-02-12 21:52:16 UTC
Waydroid does run with the above images, thanks.

Comment 29 Alessandro Astone 2025-02-12 22:52:25 UTC
And just to exclude other differences between my current build environment and the build servers, could you verify that this build *doesn't* work?

https://sourceforge.net/projects/aleasto-lineageos/files/waydroid-x86-bad.tar.gz/download

Comment 30 Andre Robatino 2025-02-12 23:04:50 UTC
The images in waydroid-x86-bad.tar.gz do NOT work - same behavior as with lineage 20250125, Waydroid never comes up graphically and dmesg is spammed with the same messages.

Comment 31 Jeffrey Walton 2025-02-12 23:23:05 UTC
(In reply to Alessandro Astone from comment #27)
> > I see from https://developer.android.com/ndk/guides/abis.html that the x86 images aren't supposed to require SSE4 at all.
> 
> Unfortunately that is wrong. AOSP by default compiles libc with SSE4.1
> instructions (but we patched those out). It should be correct instead that
> SSE4.2 is not used.

This sounds like a AOSP bug.

If you have an appetite for it, then file a bug with Android. AOSP bug tracker is located at <https://issuetracker.google.com>.

Comment 32 Alessandro Astone 2025-02-13 17:34:35 UTC
(In reply to Jeffrey Walton from comment #31)
> (In reply to Alessandro Astone from comment #27)
> > > I see from https://developer.android.com/ndk/guides/abis.html that the x86 images aren't supposed to require SSE4 at all.
> > 
> > Unfortunately that is wrong. AOSP by default compiles libc with SSE4.1
> > instructions (but we patched those out). It should be correct instead that
> > SSE4.2 is not used.
> 
> This sounds like a AOSP bug.
> 
> If you have an appetite for it, then file a bug with Android. AOSP bug
> tracker is located at <https://issuetracker.google.com>.

Oh google knows, they decided to not care:
https://android-review.googlesource.com/c/platform/bionic/+/1033638
https://android-review.googlesource.com/c/platform/bionic/+/2430852

(In reply to Andre Robatino from comment #30)
> The images in waydroid-x86-bad.tar.gz do NOT work - same behavior as with
> lineage 20250125, Waydroid never comes up graphically and dmesg is spammed
> with the same messages.

Thank you! Fix pushed at https://github.com/waydroid/android_device_waydroid_waydroid/commit/03d860d141c096377c1a8694666dc90a5a3abf21
It will be included in the next OTA update, this weekend.

Comment 33 Andre Robatino 2025-03-08 16:58:06 UTC
Just an observation - I've found that on my laptop that could use the x86_64 images, the x86 images, in addition to being smaller, seem to be just as fast, there's no obvious difference. So I'm using the x86 images on the laptop as well (set up as custom images). Do other people see a performance improvement sufficient to justify generating the additional x86_64 images?

Comment 34 Alessandro Astone 2025-03-09 22:17:54 UTC
You could ask the same about any other piece of software...

I'll be honest in saying that I don't have a good answer.


Note You need to log in before you can comment on or make changes to this bug.