Bug 2188903

Summary: Software silently fails to upgrade properly to F38 because btrfs having no data blocks left
Product: [Fedora] Fedora Reporter: Yannick Defais <sevmek>
Component: gnome-softwareAssignee: Milan Crha <mcrha>
Status: CLOSED UPSTREAM QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 37CC: gnome-sig, mcrha, rhughes
Target Milestone: ---Keywords: Desktop, Upgrades
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-04-25 12:07:10 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
/var/lib/PackageKit/offline-update-competed none

Description Yannick Defais 2023-04-23 08:11:11 UTC
I upgraded Fedora 37 to 38 using the recommended way with Software. There was 30 Gb of free space as reported by Gnome File. It did its things, but the upgrade itself took a very long time (hours), not like usual (minutes). Thus I went to do something else (you know: the very big coffee)
When I went back, the system had rebooted and were waiting for my pass-phrase. The Gnome desktop opened and it seems everything was done well.
But it seems too familiar, I expected some changes I could not find. I finally went to Gnome property -> about to find I was using "Fedora 38" with "Gnome 43" !!

The upgrade went wrong and I had no error message !

Previously I had trouble updating the system, with errors saying there were no space left in /, or saying stuff like "cpio: rename failed". I had hard time freeing space: even it there were 30 Gb free on the system (/ and /home on the same SSD using BTRFS), the update process complained about no space available. This has the side effect of maybe not having the security updates in a timely manner. And the system took very long time to try updating.

The issue here is BTRFS not only need free space on the device, but also free blocks as reported by this command:
$ sudo btrfs filesystem usage /
[sudo] Mot de passe de yannick : 
Overall:
    Device size:		 236.87GiB
    Device allocated:		 236.87GiB
    Device unallocated:		      OGiB

If the system is in this situation, having the same amount in "Device size" and "Device allocated" and 0 Device unallocated, then the update and the upgrade will fail, kind of, because some part will partially be done and some other will fail.

BTRFS being the default now, this seems important to properly deal with this situation.

IMHO, the issue is the lack of Btrfs integration in the desktop: you'll wont get proper information about your disk being full. You'll get error reporting no space left on /, but you can free multiple Gb, it will still fail.

I solved this with this command, ranking up step by step (freeing up data blocks):
$ sudo btrfs balance start -dusage=0 /
$ sudo btrfs balance start -dusage=1 /
$ sudo btrfs balance start -dusage=5 /
$ sudo btrfs balance start -dusage=10 /
$ sudo btrfs balance start -dusage=15 /
$ sudo btrfs balance start -dusage=20 /
$ sudo btrfs balance start -dusage=25 /
$ sudo btrfs balance start -dusage=30 /

After that, Btrfs had some room to fill data in the SSD:
$ sudo btrfs filesystem usage /
Overall:
    Device size:		 236.87GiB
    Device allocated:		 227.87GiB
    Device unallocated:		   9.00GiB

Then, I could do a manual update :
$ sudo dnf update
There was more than 1000 packages to update! The system felt snappy again.

After that, there was still package form F37 in the system (about 300), thus I had to do a distro-sync:
$ sudo dnf distro-sync

Hopefully, the system is now in good shape.

Reproducible: Always

Steps to Reproduce:
1. Use BTRFS for /
2. Make sure there is no data block left empty in the device
3. Try to upgrade/update
Actual Results:  
Some package will not updade/the upgrade will be stuck with package from the previous version (like the previous gnome)

Expected Results:  
Gnome Software should provide some guidance: properly point at the issue, refusing to perform the update/upgrade, and either deal with it by itself or point to manual intervention from the user to solve the issue.

Comment 1 Milan Crha 2023-04-25 06:36:19 UTC
Thanks for a bug report. I do not think the gnome-software should ever check anything about underlying file system format. That BTRFS reports a lot of free space and then refuses to save a few KB file is a bug of the BRTFS, giving the users wrong impression (and quite bad experience, to be honest). Anyway, this bug is not about BTRFS.

By any chance, do you have any files in /var/lib/PackageKit/, please? If so, which are they? Is any of them called `offline-update-competed`? Could you upload it, if is exists, please? It may contain the error message from the upgrade.

Comment 2 Milan Crha 2023-04-25 06:41:46 UTC
Corresponding upstream bug:
https://gitlab.gnome.org/GNOME/gnome-software/-/issues/666

Comment 3 Yannick Defais 2023-04-25 08:15:47 UTC
ls -l /var/lib/PackageKit/
total 12960
-rw-r--r--. 1 root root     2758 23 avril 18:31 offline-update-competed
-rw-r--r--. 1 root root 13266944 25 avril 09:43 transactions.db

I think there was another update since the upgrade...

Comment 4 Yannick Defais 2023-04-25 08:16:27 UTC
Created attachment 1959742 [details]
/var/lib/PackageKit/offline-update-competed

Comment 5 Milan Crha 2023-04-25 12:07:10 UTC
(In reply to Yannick Defais from comment #3)
> I think there was another update since the upgrade...

Thank you for the update. You are right, there had been done a successful update meanwhile, overwriting any information about the previously failed update.

If you do not mind, I'm closing this in favour of the upstream bug (mentioned above). It's supposed to extract the information from the failed update and show it to the user after start.