Bug 2088878

Summary: Silverblue: Failed to create file ?/var/run/dnf-metadata.lock.9ONHM1?: Permission denied
Product: [Fedora] Fedora Reporter: Yajo <yajo.sk8>
Component: gnome-softwareAssignee: Milan Crha <mcrha>
Status: CLOSED NEXTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: low    
Version: 36CC: adam, daniel.mach, fedora, gnome-sig, jmracek, jrohel, kanru, kxra, mblaha, mcrha, pkratoch, rhughes, rpm-software-management
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: gnome-software-44.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-04 02:43:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Failing gnome-software none

Description Yajo 2022-05-21 08:19:42 UTC
Created attachment 1881775 [details]
Failing gnome-software

Description of problem:
Gnome Software is unusable.

Version-Release number of selected component (if applicable):
rpm-ostree-2022.9-1.fc36.x86_64
gnome-software-42.1-1.fc36.x86_64


How reproducible: always


Steps to Reproduce:
1. Open it.

Actual results: See attachment.


Expected results: It should work.


Additional info:
Before this, I was having bug 2083715. Now that one's fixed, and after following bug 2083715 comment 24 instructions, this new failure started happening. Renders gnome-software still unusable.

Logs:

> gnome-software --quit

> env LANG=C gnome-software
08:12:54:0566 Gs  /etc/PackageKit/Vendor.conf file not found
08:12:54:0826 Gs  No AppStream data, try 'make install-sample-data' in data/
08:12:55:0528 Gs  Only 0 apps for popular list, hiding
08:12:55:0529 Gs  failed to get featured apps: no apps to show
08:12:55:0529 Gs  Only 0 apps for recent list, hiding
08:12:55:0954 Gs  Failed to get system: Failed to refine '*/*/*/system/*': failed to obtain lock 'metadata': Failed to create file ?/var/run/dnf-metadata.lock.N1ZFM1?: Permission denied
08:12:56:0065 flatpak Warning: Treating remote fetch error as non-fatal since runtime/net.lutris.Lutris.Runner.Wine/x86_64/beta is already installed: No such ref 'runtime/net.lutris.Lutris.Runner.Wine/x86_64/beta' in remote flathub-beta
08:12:57:0052 Gs  failed to get installed apps: failed to obtain lock 'metadata': Failed to create file ?/var/run/dnf-metadata.lock.9ONHM1?: Permission denied
08:12:58:0863 Gs  updates-shell: failed to get updates: failed to obtain lock 'metadata': Failed to create file ?/var/run/dnf-metadata.lock.ML1DM1?: Permission denied

Comment 1 Yajo 2022-05-21 08:21:00 UTC
I gues this can help too:

> rpm-ostree status -bv
State: idle
AutomaticUpdates: disabled
BootedDeployment:
● fedora:fedora/36/x86_64/silverblue
                   Version: 36.20220516.0 (2022-05-16T00:43:32Z)
                BaseCommit: e4dbe1f50a6f5fb0be04b7979c66da0fedf3b9e1d1ea8bec30684f46935cc229
                            ├─ repo-0 (2022-05-04T21:16:11Z)
                            ├─ repo-1 (2022-05-16T00:16:44Z)
                            └─ repo-2 (2022-05-16T00:18:39Z)
                    Commit: e072c47e02f70a20b7dbefe009d3466c7e42602ff0561f05eb30de82d07a7c43
                            ├─ fedora-cisco-openh264 (2022-04-07T16:52:38Z)
                            ├─ fedora-modular (2022-05-04T21:12:01Z)
                            ├─ updates-modular (2022-05-16T00:18:23Z)
                            ├─ updates (2022-05-16T01:00:22Z)
                            ├─ fedora (2022-05-04T21:16:11Z)
                            ├─ rpmfusion-free-updates (2022-05-13T10:06:59Z)
                            ├─ rpmfusion-free (2022-05-04T04:48:11Z)
                            ├─ rpmfusion-nonfree-updates (2022-05-13T10:33:29Z)
                            ├─ rpmfusion-nonfree (2022-05-04T05:11:55Z)
                            ├─ rpmfusion-nonfree-steam (2022-02-13T17:48:12Z)
                            ├─ gitlab.com_paulcarroty_vscodium_repo (2022-05-16T01:01:38Z)
                            └─ updates-archive (2022-05-16T01:26:27Z)
                    Staged: no
                 StateRoot: fedora
              GPGSignature: 1 signature
                            Signature made lun 16 may 2022 01:43:39 using RSA key ID 999F7CBF38AB71F4
                            Good signature from "Fedora <fedora-36-primary>"
           LayeredPackages: codium direnv faac fdk-aac ffmpeg fish git-subrepo gnome-boxes
                            gnome-tweaks gstreamer1-libav gstreamer1-plugin-openh264
                            gstreamer1-plugins-bad-free-extras
                            gstreamer1-plugins-bad-free-fluidsynth
                            gstreamer1-plugins-bad-free-wildmidi
                            gstreamer1-plugins-bad-free-zbar
                            gstreamer1-plugins-bad-freeworld gstreamer1-plugins-entrans
                            gstreamer1-plugins-fc gstreamer1-plugins-good-extras
                            gstreamer1-plugins-ugly gtk-v4l hunspell-es langpacks-es
                            libreoffice ltunify nautilus-python openssh-askpass openssl
                            pipewire-codec-aptx podman-docker pre-commit realtime-setup
                            steam totem v4l-utils VirtualBox
             LocalPackages: nix-multi-user-2.8.0-1.x86_64 rpmfusion-free-release-36-1.noarch
                            rpmfusion-nonfree-release-36-1.noarch xow-0.4-1.fc32.x86_64

Comment 2 Adam Israel 2022-05-21 11:59:37 UTC
This looks similar to the issue I described in https://bugzilla.redhat.com/show_bug.cgi?id=2083715#c30

Check your gitlab.com_paulcarroty_vscodium_repo repo in /etc/yum.repos.d/. The default instructions for setting that up say to set `repo_gpgcheck=1`, which will cause the exact  error you're seeing. Change that to `repo_gpgcheck=0`.

Then refresh the metadata:

```
rpm-ostree cleanup --repomd
rpm-ostree refresh-md
rm -rf ~/.cache/gnome-software
```

And re-run `gnome-software`.

If it still doesn't work, `grep "repo_gpgcheck=1" /etc/yum.repos.d/*` and see if any other repos have repo_gpgcheck enabled and, if so, disable those.

Comment 3 Yajo 2022-05-22 06:29:45 UTC
Ahm indeed that unlocked the situation. Thanks!

Isn't that lowering security? Is this a fix or just a workaround?

Comment 4 Adam Israel 2022-05-22 14:14:42 UTC
Disabling the gpg signature check of the repodata seems like a step down security-wise, yes, but none of the Fedora repos (Silverblue and Workstation) have it enabled by default.

I suspect this is just a workaround for a bug in or limitation of librepo or libdnf, based on the logs I posted in 2083715.

Comment 5 Milan Crha 2022-05-23 06:43:29 UTC
Let's move this to the libdnf, it's failing there while trying to do something crucial with an unprivileged user.

Comment 6 Lukáš Hrázký 2022-05-23 13:15:37 UTC
Gnome Software uses packagekit, which should be a privileged process. But I'm not familiar with any Silverblue-specific setup which is probably going to be quite particular. I'd first check whether dnf really runs under an unprivileged user (and why), and then also the cachedir and persistdir with which dnf is configured.

Comment 7 Milan Crha 2022-05-23 13:35:52 UTC
(In reply to Lukáš Hrázký from comment #6)
> Gnome Software uses packagekit, which should be a privileged process. But
> I'm not familiar with any Silverblue-specific setup which is probably going
> to be quite particular. I'd first check whether dnf really runs under an
> unprivileged user (and why), and then also the cachedir and persistdir with
> which dnf is configured.

You are right the GNOME Software uses PackageKit, but not on the Silverblue. The Silverblue uses rpm-ostree in combination with direct libdnf calls.

The https://bugzilla.redhat.com/show_bug.cgi?id=2083715#c5 contains a little test program, which extracts what the gnome-software does when getting list of configured repos. It can reproduce the problem from that bug, specifically the `dnf_context_setup_sack_with_flags (context, state, DNF_CONTEXT_SETUP_SACK_FLAG_SKIP_RPMDB, &error)` can fail when it tries to refresh the cache, because the process is running as a regular user (that bug is about fighting on the cache data version, where the rpm-ostree used too old bundled libdnf).

This particular error (failed write into /var/run/...) is possibly caused by a different libdnf function.

The only thing the GNOME Software can do about this is to not call functions, which can regenerate the cache, or (as those functions are required for some things) it should use flags to avoid the cache re-create, but you can argue, in the light of the bug #2083715, that the cache refresh is required to work with accurate data, which I agree with.

As I understand this problem, and I can be wrong, the rpm-ostree (with root privileges) tries to refresh repo information, but it fails due to some issue with the repo. Then there comes the gnome-software (as a regular user), only wanting to read the data, but the libdnf recognizes data/cache mismatch, wants to refresh the cache first, but it fails on the permissions. The rpm-ostree call cannot fix it, because the repo is broken, but the libdnf call, made from the gnome-software, doesn't know that and tries the same as the rpm-ostree, just failing on privileges, thus a bit sooner than the rpm-ostree.

Comment 8 Grey Nicholson 2022-05-25 11:21:58 UTC
When running `grep "repo_gpgcheck=1" /etc/yum.repos.d/*` as in #c2 to identify potentially-problematic additional repos, my installation of Silverblue 36 returned:

```
/etc/yum.repos.d/fedora-cisco-openh264.repo:repo_gpgcheck=1
```

I believe this is one of the additional repositories that Fedora suggests (during install and when running Software for the first time), and therefore this may affect users who haven't gone “off-piste” by installing unsupported third-party repos.

Comment 9 Grey Nicholson 2022-05-25 11:26:56 UTC
(In reply to Grey Nicholson from comment #8)
> /etc/yum.repos.d/fedora-cisco-openh264.repo:repo_gpgcheck=1

Looking at the contents of this file, it mentions Fedora 33, and indeed this installation has been upgraded from Silverblue 33 (and earlier).

So this specific problem *may* only affect installations that opted into the additional repos at Silverblue version 33 or less, and then upgraded all the way up to 36.

Comment 10 Yajo 2022-05-25 19:29:17 UTC
These are the contents of that file in my case. I don't see any F33 mentions:

[fedora-cisco-openh264]
name=Fedora $releasever openh264 (From Cisco) - $basearch
metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-$releasever&arch=$basearch
type=rpm
enabled=1
metadata_expire=14d
repo_gpgcheck=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
skip_if_unavailable=True

[fedora-cisco-openh264-debuginfo]
name=Fedora $releasever openh264 (From Cisco) - $basearch - Debug
metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-debug-$releasever&arch=$basearch
type=rpm
enabled=1
metadata_expire=14d
repo_gpgcheck=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
skip_if_unavailable=True

Comment 11 Yajo 2022-11-03 21:31:11 UTC
After another fresh Silverblue 36 install, I get the same failure. This time, the file that triggers the bug is:

> cat /etc/yum.repos.d/vscodium.repo
[gitlab.com_paulcarroty_vscodium_repo]
name=download.vscodium.com
baseurl=https://download.vscodium.com/rpms/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://gitlab.com/paulcarroty/vscodium-deb-rpm-repo/-/raw/master/pub.gpg
metadata_expire=1h

Setting repo_gpgcheck=0 in that file resurrects GNOME Software.

Comment 12 Milan Crha 2022-11-04 02:43:01 UTC
I believe the `repo_gpgcheck=0` change is just a coincident.

There is [1], for which exists a change on the rpm-ostree side and a proposed fix for the gnome-software. An intention is to have this fixed for 44.x series of the gnome-software, which is far away, but there's no option due t added API on the rpm-ostree side.

[1] https://gitlab.gnome.org/GNOME/gnome-software/-/issues/1793

Comment 13 Milan Crha 2022-11-21 08:09:12 UTC
*** Bug 2144034 has been marked as a duplicate of this bug. ***