Bug 2035611 - Gracefully handle failure of `flatpak remotes` due to corrupted Flatpak directories
Summary: Gracefully handle failure of `flatpak remotes` due to corrupted Flatpak direc...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: fedora-third-party
Version: 35
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Owen Taylor
QA Contact:
URL: https://retrace.fedoraproject.org/faf...
Whiteboard: abrt_hash:8438ffa1339e96c5b4bb2b506e6...
: 2019633 2022256 2061153 2072416 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-12-25 15:35 UTC by Michael Xavier
Modified: 2022-04-20 23:29 UTC (History)
7 users (show)

Fixed In Version: fedora-third-party-0.10-1.fc36
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-04-20 23:29:33 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
File: backtrace (46.17 KB, text/plain)
2021-12-25 15:35 UTC, Michael Xavier
no flags Details
File: core_backtrace (12.05 KB, text/plain)
2021-12-25 15:35 UTC, Michael Xavier
no flags Details
File: cpuinfo (2.93 KB, text/plain)
2021-12-25 15:35 UTC, Michael Xavier
no flags Details
File: dso_list (740 bytes, text/plain)
2021-12-25 15:35 UTC, Michael Xavier
no flags Details
File: environ (1.41 KB, text/plain)
2021-12-25 15:35 UTC, Michael Xavier
no flags Details
File: exploitable (112 bytes, text/plain)
2021-12-25 15:35 UTC, Michael Xavier
no flags Details
File: limits (1.29 KB, text/plain)
2021-12-25 15:35 UTC, Michael Xavier
no flags Details
File: maps (3.96 KB, text/plain)
2021-12-25 15:35 UTC, Michael Xavier
no flags Details
File: mountinfo (3.18 KB, text/plain)
2021-12-25 15:35 UTC, Michael Xavier
no flags Details
File: open_fds (4.24 KB, text/plain)
2021-12-25 15:35 UTC, Michael Xavier
no flags Details
File: proc_pid_status (1.39 KB, text/plain)
2021-12-25 15:35 UTC, Michael Xavier
no flags Details
File: var_log_messages (222 bytes, text/plain)
2021-12-25 15:35 UTC, Michael Xavier
no flags Details

Description Michael Xavier 2021-12-25 15:35:38 UTC
Description of problem:
Al entrar a repositorio de software automaticamente se cierra la tienda de software, yo creo que puede ser por instalar los repositorios de rpmfusion free and not-free

Version-Release number of selected component:
gnome-software-41.2-1.fc35

Additional info:
reporter:       libreport-2.15.2
backtrace_rating: 4
cgroup:         0::/user.slice/user-1000.slice/user/app.slice/app-gnome-gnome\x2dsoftware\x2dservice-2454.scope
cmdline:        /usr/bin/gnome-software --gapplication-service
crash_function: fedora_third_party_list_repos_done_cb
executable:     /usr/bin/gnome-software
journald_cursor: s=31428979407146b19a67faedd4331689;i=112c4;b=f745123c45c24c3592d19fab23dfdd9a;m=103189cd;t=5d3f2c37b81d4;x=8d837c0a306088e6
kernel:         5.15.10-200.fc35.x86_64
rootdir:        /
runlevel:       N 5
type:           CCpp
uid:            1000

Truncated backtrace:
Thread no. 1 (6 frames)
 #0 fedora_third_party_list_repos_done_cb at ../src/gs-repos-dialog.c:597
 #1 g_task_return_now at ../gio/gtask.c:1219
 #2 complete_in_idle_cb at ../gio/gtask.c:1233
 #6 g_main_context_iterate.constprop.0 at ../glib/gmain.c:4175
 #7 g_main_context_iteration at ../glib/gmain.c:4240
 #8 g_application_run at ../gio/gapplication.c:2569

Comment 1 Michael Xavier 2021-12-25 15:35:42 UTC
Created attachment 1847775 [details]
File: backtrace

Comment 2 Michael Xavier 2021-12-25 15:35:43 UTC
Created attachment 1847776 [details]
File: core_backtrace

Comment 3 Michael Xavier 2021-12-25 15:35:45 UTC
Created attachment 1847777 [details]
File: cpuinfo

Comment 4 Michael Xavier 2021-12-25 15:35:46 UTC
Created attachment 1847778 [details]
File: dso_list

Comment 5 Michael Xavier 2021-12-25 15:35:47 UTC
Created attachment 1847779 [details]
File: environ

Comment 6 Michael Xavier 2021-12-25 15:35:48 UTC
Created attachment 1847780 [details]
File: exploitable

Comment 7 Michael Xavier 2021-12-25 15:35:49 UTC
Created attachment 1847781 [details]
File: limits

Comment 8 Michael Xavier 2021-12-25 15:35:50 UTC
Created attachment 1847782 [details]
File: maps

Comment 9 Michael Xavier 2021-12-25 15:35:51 UTC
Created attachment 1847783 [details]
File: mountinfo

Comment 10 Michael Xavier 2021-12-25 15:35:52 UTC
Created attachment 1847784 [details]
File: open_fds

Comment 11 Michael Xavier 2021-12-25 15:35:53 UTC
Created attachment 1847785 [details]
File: proc_pid_status

Comment 12 Michael Xavier 2021-12-25 15:35:55 UTC
Created attachment 1847786 [details]
File: var_log_messages

Comment 13 Milan Crha 2022-01-03 10:51:35 UTC
Thanks for a bug report. The backtrace suggests there failed to call `fedora-third-party` binary for some reason, but there was not set any error (the reason is unknown), which resulted in this crash.

Could you try to run a terminal and in it the below command, please?

   $ fedora-third-party list

It lists currently configured repositories for the Fedora 3rd-party repositories.

Comment 14 Michael Xavier 2022-01-25 19:01:50 UTC
Similar problem has been detected:

CLick on software repository in gnome software

reporter:       libreport-2.15.2
backtrace_rating: 4
cgroup:         0::/user.slice/user-1000.slice/user/app.slice/app-gnome-gnome\x2dsoftware\x2dservice-2440.scope
cmdline:        /usr/bin/gnome-software --gapplication-service
crash_function: fedora_third_party_list_repos_done_cb
executable:     /usr/bin/gnome-software
journald_cursor: s=f3d94750f91444cfbd0e27d14854caad;i=3a8d6;b=0bfe7d1fca034e69aa6752d72b9ff1dd;m=1599dce2;t=5d66c5ee58dbe;x=6fab4741a81aea0a
kernel:         5.15.16-200.fc35.x86_64
package:        gnome-software-41.3-1.fc35
reason:         gnome-software killed by SIGSEGV
rootdir:        /
runlevel:       N 5
type:           CCpp
uid:            1000

Comment 15 Michael Xavier 2022-01-25 19:06:51 UTC
$ fedora-third-party list
error: Mientras se abría el repositorio /var/lib/flatpak/repo: opening repo: opendir(objects): No existe el fichero o el directorio
Traceback (most recent call last):
  File "/usr/bin/fedora-third-party", line 33, in <module>
    sys.exit(load_entry_point('fedora-third-party==0.8', 'console_scripts', 'fedora-third-party')())
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/fedora_third_party/cli.py", line 101, in list
    repositories = [r for r in cfg.list_repositories() if not r.is_hidden()]
  File "/usr/lib/python3.10/site-packages/fedora_third_party/cli.py", line 101, in <listcomp>
    repositories = [r for r in cfg.list_repositories() if not r.is_hidden()]
  File "/usr/lib/python3.10/site-packages/fedora_third_party/repository.py", line 92, in is_hidden
    remote = self.get_existing_remote()
  File "/usr/lib/python3.10/site-packages/fedora_third_party/repository.py", line 88, in get_existing_remote
    return next((r for r in get_flatpak_remotes() if r.name == self.name), None)
  File "/usr/lib/python3.10/site-packages/fedora_third_party/repository.py", line 60, in get_flatpak_remotes
    output = subprocess.check_output([
  File "/usr/lib64/python3.10/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib64/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['flatpak', 'remotes', '--system', '--columns=name,filter']' returned non-zero exit status 1.

Comment 16 Michael Xavier 2022-01-25 19:22:39 UTC
sudo rm -r /var/lib/flatpak/repo

I found the solution of my problem.

Comment 17 Milan Crha 2022-01-26 08:37:45 UTC
Thanks for the update. I'm moving this to fedora-third-party. Ideally, it should not crash in this case (which caused the crash of the gnome-software, because it did not receive GError as expected).

Comment 18 Owen Taylor 2022-04-06 15:34:59 UTC
IMO, there was a GNOME Software error here as well as a fedora-third-party bug - GNOME Software needs to handle fedora-third-party failing gracefully. Since GNOME Software isn't even checking the process exit code from fedora-third-party it has no idea whether fedora-third-party is:

 Failing with a traceback
 Failing without a traceback
 Simply listing no third-party repositories

https://gitlab.gnome.org/GNOME/gnome-software/-/issues/1644 fixed the GNOME Software crash in this case.

I *do* think that gs_fedora_third_party_switch_sync() and other cases should be checking the exit status from fedora-third-party. It makes little sense to me to treat errors that happen pre-exec (no fedora-third-party binary, or whatever) as an "error", but not treat an abnormal exit status as an "error". (g_spawn_check_wait_status() could be used.)

===

I'll look at making fedora-third-party handle the case of "corrupted Flatpak repository causes flatpak to fail" more gracefully - without a traceback and triggering a system crash report. (This is a fairly frequent reported problem on retrace.fedoraproject.org)

As far as I can determine from the above, from searching the internet, and talking to some people in the Flatpak community, the most likely scenario for getting to this state is something like:
 
 - User is low on disk space
 - User determines that /var/lib/flatpak/repo/objects is one of the biggest directories on their system
 - User decides they don't use flatpaks and deletes the directory

Unfortunately, 'flaptak repair' is not currently able to recover from this state.

Comment 19 Owen Taylor 2022-04-06 15:47:27 UTC
Repair instructions
===================

If only the objects/ directory is missing, then it's possible to repair with:

  sudo mkdir /var/lib/flatpak/repo/objects
  sudo flatpak repair

If that doesn't work, then it gets more complicated, something like (as root):

  # Remove the entire repository, including the set of Flatpak remotes
  rm -rf /var/lib/flatpak/repo
  # Add the Fedora Flatpaks 
  /usr/bin/flatpak remote-add --system --if-not-exists --title "Fedora Flatpaks" fedora oci+https://registry.fedoraproject.org
  # Add the Fedora Flathub Selection repository back by repeating opt-in / opt-out
  rm /var/lib/fedora-third-party/state 
  fedora-third-party [enable/disable]
  # Add other remotes you installed Flatpaks from (full Flathub, etc)    
  flatpak repair

Comment 20 Owen Taylor 2022-04-06 15:52:42 UTC
*** Bug 2019633 has been marked as a duplicate of this bug. ***

Comment 21 Owen Taylor 2022-04-06 15:53:49 UTC
*** Bug 2022256 has been marked as a duplicate of this bug. ***

Comment 22 Owen Taylor 2022-04-06 15:55:01 UTC
*** Bug 2061153 has been marked as a duplicate of this bug. ***

Comment 23 Owen Taylor 2022-04-06 15:59:29 UTC
*** Bug 2072416 has been marked as a duplicate of this bug. ***

Comment 24 Fedora Update System 2022-04-07 04:32:22 UTC
FEDORA-2022-1ed936a3ac has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-1ed936a3ac

Comment 25 gsckorohod 2022-04-07 13:33:03 UTC
(In reply to Owen Taylor from comment #7)
> Can you please answer the following questions in a comment on bug 2035611:
> 
>  * Does 'flatpak remotes --system --columns=name,filter' fail if you run it
> from the command line? 
>  * If so:
>   - what is the output
>   - do you remember doing any manual manipulation of /var/lib/flatpaks
> (removing files from it to save space)
>   - are the repair instructions from
> https://bugzilla.redhat.com/show_bug.cgi?id=2035611#c18 helpful?
> 
> Many thanks!
> 
> *** This bug has been marked as a duplicate of bug 2035611 ***

Hi! The command works, here is the output:

Name     Filter
fedora  -
flathub -

I did not do any manipulations with /var/lib/flatpaks. At least not manually, and I don't remember running anything that could've affected it.

I found the suggestion to create /var/lib/flatpak/repo/objects manually and it worked, I installed a few flatpak apps from the command line without a problem. After a reboot, opening .flatpakref files via Gnome App Centre also works

Comment 26 gsckorohod 2022-04-07 13:33:58 UTC
Thanks for the instructions!

Comment 27 Fedora Update System 2022-04-07 18:01:20 UTC
FEDORA-2022-1ed936a3ac has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-1ed936a3ac`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-1ed936a3ac

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 28 Milan Crha 2022-04-08 07:12:10 UTC
(In reply to Owen Taylor from comment #18)
> I *do* think that gs_fedora_third_party_switch_sync() and other cases should
> be checking the exit status from fedora-third-party. It makes little sense
> to me to treat errors that happen pre-exec (no fedora-third-party binary, or
> whatever) as an "error", but not treat an abnormal exit status as an
> "error". (g_spawn_check_wait_status() could be used.)

I opened the following merge request for it:
https://gitlab.gnome.org/GNOME/gnome-software/-/merge_requests/1316

Comment 29 Fedora Update System 2022-04-20 23:29:33 UTC
FEDORA-2022-1ed936a3ac has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.