Bug 2188903 - Software silently fails to upgrade properly to F38 because btrfs having no data blocks left
Summary: Software silently fails to upgrade properly to F38 because btrfs having no da...
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Fedora
Classification: Fedora
Component: gnome-software
Version: 37
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Milan Crha
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-04-23 08:11 UTC by Yannick Defais
Modified: 2023-04-25 12:07 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-04-25 12:07:10 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
/var/lib/PackageKit/offline-update-competed (2.69 KB, text/plain)
2023-04-25 08:16 UTC, Yannick Defais
no flags Details


Links
System ID Private Priority Status Summary Last Updated
GNOME Gitlab GNOME gnome-software issues 666 0 None closed Offline updates are failing with no error message 2023-04-25 06:41:46 UTC

Description Yannick Defais 2023-04-23 08:11:11 UTC
I upgraded Fedora 37 to 38 using the recommended way with Software. There was 30 Gb of free space as reported by Gnome File. It did its things, but the upgrade itself took a very long time (hours), not like usual (minutes). Thus I went to do something else (you know: the very big coffee)
When I went back, the system had rebooted and were waiting for my pass-phrase. The Gnome desktop opened and it seems everything was done well.
But it seems too familiar, I expected some changes I could not find. I finally went to Gnome property -> about to find I was using "Fedora 38" with "Gnome 43" !!

The upgrade went wrong and I had no error message !

Previously I had trouble updating the system, with errors saying there were no space left in /, or saying stuff like "cpio: rename failed". I had hard time freeing space: even it there were 30 Gb free on the system (/ and /home on the same SSD using BTRFS), the update process complained about no space available. This has the side effect of maybe not having the security updates in a timely manner. And the system took very long time to try updating.

The issue here is BTRFS not only need free space on the device, but also free blocks as reported by this command:
$ sudo btrfs filesystem usage /
[sudo] Mot de passe de yannick : 
Overall:
    Device size:		 236.87GiB
    Device allocated:		 236.87GiB
    Device unallocated:		      OGiB

If the system is in this situation, having the same amount in "Device size" and "Device allocated" and 0 Device unallocated, then the update and the upgrade will fail, kind of, because some part will partially be done and some other will fail.

BTRFS being the default now, this seems important to properly deal with this situation.

IMHO, the issue is the lack of Btrfs integration in the desktop: you'll wont get proper information about your disk being full. You'll get error reporting no space left on /, but you can free multiple Gb, it will still fail.

I solved this with this command, ranking up step by step (freeing up data blocks):
$ sudo btrfs balance start -dusage=0 /
$ sudo btrfs balance start -dusage=1 /
$ sudo btrfs balance start -dusage=5 /
$ sudo btrfs balance start -dusage=10 /
$ sudo btrfs balance start -dusage=15 /
$ sudo btrfs balance start -dusage=20 /
$ sudo btrfs balance start -dusage=25 /
$ sudo btrfs balance start -dusage=30 /

After that, Btrfs had some room to fill data in the SSD:
$ sudo btrfs filesystem usage /
Overall:
    Device size:		 236.87GiB
    Device allocated:		 227.87GiB
    Device unallocated:		   9.00GiB

Then, I could do a manual update :
$ sudo dnf update
There was more than 1000 packages to update! The system felt snappy again.

After that, there was still package form F37 in the system (about 300), thus I had to do a distro-sync:
$ sudo dnf distro-sync

Hopefully, the system is now in good shape.

Reproducible: Always

Steps to Reproduce:
1. Use BTRFS for /
2. Make sure there is no data block left empty in the device
3. Try to upgrade/update
Actual Results:  
Some package will not updade/the upgrade will be stuck with package from the previous version (like the previous gnome)

Expected Results:  
Gnome Software should provide some guidance: properly point at the issue, refusing to perform the update/upgrade, and either deal with it by itself or point to manual intervention from the user to solve the issue.

Comment 1 Milan Crha 2023-04-25 06:36:19 UTC
Thanks for a bug report. I do not think the gnome-software should ever check anything about underlying file system format. That BTRFS reports a lot of free space and then refuses to save a few KB file is a bug of the BRTFS, giving the users wrong impression (and quite bad experience, to be honest). Anyway, this bug is not about BTRFS.

By any chance, do you have any files in /var/lib/PackageKit/, please? If so, which are they? Is any of them called `offline-update-competed`? Could you upload it, if is exists, please? It may contain the error message from the upgrade.

Comment 2 Milan Crha 2023-04-25 06:41:46 UTC
Corresponding upstream bug:
https://gitlab.gnome.org/GNOME/gnome-software/-/issues/666

Comment 3 Yannick Defais 2023-04-25 08:15:47 UTC
ls -l /var/lib/PackageKit/
total 12960
-rw-r--r--. 1 root root     2758 23 avril 18:31 offline-update-competed
-rw-r--r--. 1 root root 13266944 25 avril 09:43 transactions.db

I think there was another update since the upgrade...

Comment 4 Yannick Defais 2023-04-25 08:16:27 UTC
Created attachment 1959742 [details]
/var/lib/PackageKit/offline-update-competed

Comment 5 Milan Crha 2023-04-25 12:07:10 UTC
(In reply to Yannick Defais from comment #3)
> I think there was another update since the upgrade...

Thank you for the update. You are right, there had been done a successful update meanwhile, overwriting any information about the previously failed update.

If you do not mind, I'm closing this in favour of the upstream bug (mentioned above). It's supposed to extract the information from the failed update and show it to the user after start.


Note You need to log in before you can comment on or make changes to this bug.