Description of problem: Flatpak is writing a lot of data to /var/lib/flatpak/appstream. I'm told this directory should just contain metadata, but somehow the disk usage there grows over time (in my case to 6.7 GB). Version-Release number of selected component (if applicable): 1.12.2-1 How reproducible: I've only seen this on two systems, but I'm told others have experienced similar behavior (see https://ask.fedoraproject.org/t/flatpak-using-much-more-storage-space-than-installed-packages/18160/14) Steps to Reproduce: 1. Install flatpak 3. Using flatpak for a period of time, installing and upgrading packages occasionally 4. Check disk usage of flatpak directories (e.g. with `cd /var/lib/flatpak; sudo du -hs repo app runtime appstream exports oci overrides`) Actual results: `cd /var/lib/flatpak; sudo du -hs repo app runtime appstream exports oci overrides` shows a large accumulation of files in /var/lib/flatpak/appstream (about 6.7 GB): $ sudo du -hs repo app runtime appstream exports oci overrides 3.2G repo 2.2M app 482M runtime 6.7G appstream 360K exports 2.3M oci 8.0K overrides It looks like most of the disk usage within appstream is in the appstream/flathub/x86_64 subdirectory: $ sudo du -hs repo appstream/flathub/x86_64 3.2G repo 6.6G appstream/flathub/x86_64 appstream/flathub/x86_64 contains a lot of hidden (.) subdirectories, 736 in total. Individually don't take up much space (the largest is only 19 MB), but apparently they add up. Expected results: As I understand it, most of the disk usage should be in /var/lib/flatpak/repo with comparatively little in the other subdirectories of /var/lib/flatpak. Additional info: This is a system that has been upgraded through several versions of Fedora, so some of the files may have been left by earlier versions of Flatpak.
FWIW I am seeing the same thing on several of my systems — .-prefixed directories under /var/lib/flatpak/appstream/flathub/x86_64 taking up hundreds of megabytes.
I renamed /var/lib/flatpak/appstream just before posting this bug (still have the original in case it's useful for diagnosing the problem). In the 29 days since, the new appstream directory has accumulated to 941 MB. As before, most of this disk usage (938 MB) is in the flathub subdirectory.
In today's Prioritized Bugs meeting, we agreed to accept this as a Prioritized Bug. https://meetbot.fedoraproject.org/fedora-meeting-1/2022-01-12/fedora_prioritized_bugs_and_issues.2022-01-12-16.00.log.html#l-147
Yep. $ sudo du -sh /var/lib/flatpak 8.9G /var/lib/flatpak $ sudo du -sh /var/lib/flatpak/appstream/ 3.2G /var/lib/flatpak/appstream/ $ sudo du -sh /var/lib/flatpak/appstream/* 2.3M /var/lib/flatpak/appstream/fedora 2.3M /var/lib/flatpak/appstream/fedora-testing 64K /var/lib/flatpak/appstream/firefoxdevedition-origin 1.9G /var/lib/flatpak/appstream/flathub 1.3G /var/lib/flatpak/appstream/slack-origin
(In reply to Chris Murphy from comment #4) > $ sudo du -sh /var/lib/flatpak/appstream/ > 3.2G /var/lib/flatpak/appstream/ High disk usage in /var/lib/flatpak/appstream doesn't indicate a bug by itself...you might just have a lot of flatpaks installed. Most of the stuff in appstream is just hard links to things that are also found in /var/lib/flatpak/repo. If you run the following instead: $ sudo du -sh /var/lib/flatpak/repo /var/lib/flatpak/appstream that will show you the disk usage of both repo and appstream, without double-counting anything in appstream that was already found in repo.
$ sudo du -sh /var/lib/flatpak/repo /var/lib/flatpak/appstream 5.6G /var/lib/flatpak/repo 3.1G /var/lib/flatpak/appstream 10 flatpak apps
(In reply to John Haiducek from comment #5) > (In reply to Chris Murphy from comment #4) > > $ sudo du -sh /var/lib/flatpak/appstream/ > > 3.2G /var/lib/flatpak/appstream/ > > High disk usage in /var/lib/flatpak/appstream doesn't indicate a bug by > itself...you might just have a lot of flatpaks installed. Most of the stuff > in appstream is just hard links to things that are also found in > /var/lib/flatpak/repo. If you run the following instead: Here's `/var/lib/flatpak/appstream/flathub/x86_64` on my laptop right now: ``` drwxr-xr-x. 1 root root 3150 Feb 8 09:32 . drwxr-xr-x. 1 root root 12 Oct 7 16:48 .. drwxr-xr-x. 1 root root 68 Jan 3 06:34 .13b6ec0f6ada8b8eeee420ba97f7d01fadcba3be8e5fd789002120711cef894b-C32AF1 drwxr-xr-x. 1 root root 68 Oct 28 08:55 .22de5f3845a102aa9a56bc7a0526b6a5c9601106385924e7e79ce791c549b9a5-3YHTB1 drwxr-xr-x. 1 root root 68 Oct 28 08:55 .22de5f3845a102aa9a56bc7a0526b6a5c9601106385924e7e79ce791c549b9a5-HWITB1 drwxr-xr-x. 1 root root 68 Oct 28 08:55 .22de5f3845a102aa9a56bc7a0526b6a5c9601106385924e7e79ce791c549b9a5-YUITB1 drwxr-xr-x. 1 root root 68 Oct 27 06:24 .4287995a7c65fc00e9fa749ebf10847a42e300d9f5536143d93c965a58111510-N1EWB1 drwxr-xr-x. 1 root root 68 Oct 7 16:48 .54278a8795fa0c095d02f56d3b1a0cd12ca0569bda5c2ac31926b9ef350c1d24-ad566866d9b9612f79027af9512ea23aaed50320-5VMVA1 drwxr-xr-x. 1 root root 68 Oct 7 16:48 .54278a8795fa0c095d02f56d3b1a0cd12ca0569bda5c2ac31926b9ef350c1d24-ad566866d9b9612f79027af9512ea23aaed50320-DOJVA1 drwxr-xr-x. 1 root root 68 Oct 7 16:48 .54278a8795fa0c095d02f56d3b1a0cd12ca0569bda5c2ac31926b9ef350c1d24-ad566866d9b9612f79027af9512ea23aaed50320-FKIVA1 drwxr-xr-x. 1 root root 68 Oct 7 16:48 .54278a8795fa0c095d02f56d3b1a0cd12ca0569bda5c2ac31926b9ef350c1d24-ad566866d9b9612f79027af9512ea23aaed50320-YVMVA1 drwxr-xr-x. 1 root root 68 Oct 7 16:48 .54278a8795fa0c095d02f56d3b1a0cd12ca0569bda5c2ac31926b9ef350c1d24-ad566866d9b9612f79027af9512ea23aaed50320-Z20UA1 drwxr-xr-x. 1 root root 68 Dec 1 06:25 .71674e416672ecf3368b577e0532e3c876bbc0d5f8d1621f5b5acd0a8b295708-BJXOD1 lrwxrwxrwx. 1 root root 64 Feb 8 09:32 active -> bae09f79b1604e3ffa2e421d9515deefd89e863836bf78a03770e5d8efb687c0 lrwxrwxrwx. 1 root root 105 Oct 7 16:48 .active-05A1Wp -> 54278a8795fa0c095d02f56d3b1a0cd12ca0569bda5c2ac31926b9ef350c1d24-ad566866d9b9612f79027af9512ea23aaed50320 lrwxrwxrwx. 1 root root 64 Oct 27 09:59 .active-cV2RwW -> f69a90e0033a81763b0e152f7177160a8664c9f330fd4c2dd6765c03f4908111 lrwxrwxrwx. 1 root root 64 Oct 27 09:59 .active-eKGclm -> f69a90e0033a81763b0e152f7177160a8664c9f330fd4c2dd6765c03f4908111 lrwxrwxrwx. 1 root root 64 Oct 27 06:24 .active-fEuhiQ -> 4287995a7c65fc00e9fa749ebf10847a42e300d9f5536143d93c965a58111510 lrwxrwxrwx. 1 root root 64 Oct 28 06:26 .active-hvIBME -> be447b9d003ca9a4def45a7ad541f7959b58ba069a0eaf08cfc1a8116237101d lrwxrwxrwx. 1 root root 105 Oct 7 16:48 .active-IPNqhT -> 54278a8795fa0c095d02f56d3b1a0cd12ca0569bda5c2ac31926b9ef350c1d24-ad566866d9b9612f79027af9512ea23aaed50320 lrwxrwxrwx. 1 root root 64 Oct 27 09:59 .active-jGxoA4 -> f69a90e0033a81763b0e152f7177160a8664c9f330fd4c2dd6765c03f4908111 lrwxrwxrwx. 1 root root 64 Oct 28 08:55 .active-jqwdM8 -> 22de5f3845a102aa9a56bc7a0526b6a5c9601106385924e7e79ce791c549b9a5 lrwxrwxrwx. 1 root root 64 Dec 1 06:25 .active-lO9PfZ -> 71674e416672ecf3368b577e0532e3c876bbc0d5f8d1621f5b5acd0a8b295708 lrwxrwxrwx. 1 root root 105 Oct 7 16:48 .active-pGPK5h -> 54278a8795fa0c095d02f56d3b1a0cd12ca0569bda5c2ac31926b9ef350c1d24-ad566866d9b9612f79027af9512ea23aaed50320 lrwxrwxrwx. 1 root root 64 Oct 28 08:55 .active-QtpRX8 -> 22de5f3845a102aa9a56bc7a0526b6a5c9601106385924e7e79ce791c549b9a5 lrwxrwxrwx. 1 root root 105 Oct 7 16:48 .active-RRpptz -> 54278a8795fa0c095d02f56d3b1a0cd12ca0569bda5c2ac31926b9ef350c1d24-ad566866d9b9612f79027af9512ea23aaed50320 lrwxrwxrwx. 1 root root 105 Oct 7 16:48 .active-smV4xw -> 54278a8795fa0c095d02f56d3b1a0cd12ca0569bda5c2ac31926b9ef350c1d24-ad566866d9b9612f79027af9512ea23aaed50320 lrwxrwxrwx. 1 root root 64 Oct 28 08:55 .active-tNtzxT -> 22de5f3845a102aa9a56bc7a0526b6a5c9601106385924e7e79ce791c549b9a5 lrwxrwxrwx. 1 root root 64 Jan 3 06:34 .active-Xiuxke -> 13b6ec0f6ada8b8eeee420ba97f7d01fadcba3be8e5fd789002120711cef894b drwxr-xr-x. 1 root root 68 Feb 8 09:32 bae09f79b1604e3ffa2e421d9515deefd89e863836bf78a03770e5d8efb687c0 drwxr-xr-x. 1 root root 68 Oct 28 06:26 .be447b9d003ca9a4def45a7ad541f7959b58ba069a0eaf08cfc1a8116237101d-L0TQB1 drwxr-xr-x. 1 root root 68 Oct 27 09:59 .f69a90e0033a81763b0e152f7177160a8664c9f330fd4c2dd6765c03f4908111-SF0VB1 drwxr-xr-x. 1 root root 68 Oct 27 09:59 .f69a90e0033a81763b0e152f7177160a8664c9f330fd4c2dd6765c03f4908111-SH0VB1 drwxr-xr-x. 1 root root 68 Oct 27 09:59 .f69a90e0033a81763b0e152f7177160a8664c9f330fd4c2dd6765c03f4908111-TH0VB1 -rw-r--r--. 1 root root 0 Feb 8 09:32 .timestamp ``` Most of those symlinks are broken. I think it's the dot-prefixed real directories that are the issue.
This directory is only supposed to have (in the steady state), one "active" symlink pointing to a non-dot prefixed check-summed file, and a .timestamp file. So, in the last comment: drwxr-xr-x. 1 root root 3150 Feb 8 09:32 . drwxr-xr-x. 1 root root 12 Oct 7 16:48 .. lrwxrwxrwx. 1 root root 64 Feb 8 09:32 active -> bae09f79b1604e3ffa2e421d9515deefd89e863836bf78a03770e5d8efb687c0 drwxr-xr-x. 1 root root 68 Feb 8 09:32 bae09f79b1604e3ffa2e421d9515deefd89e863836bf78a03770e5d8efb687c0 -rw-r--r--. 1 root root 0 Feb 8 09:32 .timestamp The updating of this directory happens in deploy_appstream(), here: https://github.com/flatpak/flatpak/blob/main/common/flatpak-dir.c#L4366 And basically it does: 1. get the checksum of the new dir (${newdigest}) 2. create a temp dir called .${newdigest}-XXXXXX (X are random) 3. check out the new content into the tmp fir 4. create a new tmp symlink called .active-XXXXX pointing to ${newdigest} 5. Call syncfs() to ensure everything is written to disk 6. Rename the tempdir to ${newdigest} 7. rename the tmp link to "active" (atomically replacing the old one) If all this goes right, then there should not be any leftovers. However, we can see from e.g.: drwxr-xr-x. 1 root root 68 Oct 28 08:55 .22de5f3845a102aa9a56bc7a0526b6a5c9601106385924e7e79ce791c549b9a5-3YHTB1 drwxr-xr-x. 1 root root 68 Oct 28 08:55 .22de5f3845a102aa9a56bc7a0526b6a5c9601106385924e7e79ce791c549b9a5-HWITB1 drwxr-xr-x. 1 root root 68 Oct 28 08:55 .22de5f3845a102aa9a56bc7a0526b6a5c9601106385924e7e79ce791c549b9a5-YUITB1 lrwxrwxrwx. 1 root root 64 Oct 28 08:55 .active-jqwdM8 -> 22de5f3845a102aa9a56bc7a0526b6a5c9601106385924e7e79ce791c549b9a5 lrwxrwxrwx. 1 root root 64 Oct 28 08:55 .active-QtpRX8 -> 22de5f3845a102aa9a56bc7a0526b6a5c9601106385924e7e79ce791c549b9a5 lrwxrwxrwx. 1 root root 64 Oct 28 08:55 .active-tNtzxT -> 22de5f3845a102aa9a56bc7a0526b6a5c9601106385924e7e79ce791c549b9a5 That it tried to create the directory for 22de5f3845a102aa9a56bc7a0526b6a5c9601106385924e7e79ce791c549b9a5 3 times, but something caused it to fail. Such failures should have returned an error to the caller, but unfortunately the temporary directories are left. So, the simple (and correct) fix is to make the temp dir use "g_auto(GLnxTmpDir)" so that it is automatically deleted on error. However, it would be interesting to understand what is causing these failures. Some I/O in this is constantly failing, and it would be nice to know what.
Oh, forgot in the above list: 8. delete the directory that was previously pointed to by "active".
There is a non-I/O error codepath that could happen, and that is if multiple parallel versions of this are running. Then only the first will succeed in renaming to the final name, and the others will get EEXIST from the rename. However, why would there be 3 parallel updates at the same time?
The fact that the .active symlinks are there means that any error must happen in 5-7 (i.e. after it gets created), and assuming that the errors are not from syncfs (because that should not be this common without everything being horribly bad) the error must be from the rename, so I guess the issue is that of parallel updates. Still, the fix is the same, just delete the tmpdir on any error, which is easy with g_auto(GLnxTmpDir).
I added this issue upstream: https://github.com/flatpak/flatpak/issues/4735
Fix in https://github.com/flatpak/flatpak/pull/4736
Flatpak 1.12.5 is out with the fix for this.
FEDORA-2022-1b28f1ed5e has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-1b28f1ed5e
FEDORA-2022-1b28f1ed5e has been pushed to the Fedora 35 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-1b28f1ed5e` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-1b28f1ed5e See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2022-1b28f1ed5e has been pushed to the Fedora 35 stable repository. If problem still persists, please make note of it in this bug report.