Bug 2030255

Summary: Intermittent failure of `dnf --cacheonly`
Product: [Fedora] Fedora Reporter: Jonathan Watt <jwatt>
Component: dnfAssignee: Marek Blaha <mblaha>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 35CC: ajvincent, daniel.mach, jmracek, jrohel, mblaha, mhatina, packaging-team-maint, pkratoch, rpm-software-management, vmukhame
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: dnf-4.11.1-1.fc37 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-14 13:09:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jonathan Watt 2021-12-08 10:19:45 UTC
Summary:

Would it be possible to create the 'gen' subdirectory in '/var/cache/dnf/updates-<hex-string>/repodata' up front as soon as that directory is created, and to avoid removing it while that cache is still in use?

Details:

The Mozilla Firefox build scripts have some code to "bootstrap" your local machine to install build prerequisites for devs. This code should be run regularly to ensure dependencies are up to date or else weird build failures can occur. To avoid being annoying, the script tries to avoid prompting the user for their password to `sudo dnf install` where possible by checking whether the prerequisites are all present:

https://searchfox.org/mozilla-central/rev/eea0fac2cb8cae1e5a2e38959eeaa0b7f6d23456/python/mozboot/mozboot/base.py#466,473,485-486

It does that using `dnf --cacheonly group list --installed' (without using sudo, obviously, or that would defeat the purpose of the check!)

Unfortunately, this can sometimes fail:

```
$ dnf --cacheonly group list --installed
Last metadata expiration check: 0:42:44 ago on Wed 08 Dec 2021 09:00:58 GMT.
[Errno 13] Permission denied: '/var/cache/dnf/updates-7eea87b22825bc0d/repodata/gen'
$ sudo ls -al /var/cache/dnf/updates-7eea87b22825bc0d/repodata/
total 15200
drwxr-xr-x. 1 root root      868 Dec  8 09:00 .
drwxr-xr-x. 1 root root       56 Dec  8 09:00 ..
-rw-r--r--. 1 root root    16908 Dec  8 09:00 <a-bunch-of-these>.xml.zck
-rw-r--r--. 1 root root     8306 Dec  8 09:00 repomd.xml
```

Retrying with sudo, it succeeds and the 'gen' subdirectory is created:

```
$ sudo dnf --cacheonly group list --installed
Last metadata expiration check: 0:43:02 ago on Wed 08 Dec 2021 09:00:58 GMT.
Installed Groups:
   <a-bunch-of-groups-listed>
$ sudo ls -al /var/cache/dnf/updates-7eea87b22825bc0d/repodata/
total 15200
drwxr-xr-x. 1 root root      874 Dec  8 09:44 .
drwxr-xr-x. 1 root root       56 Dec  8 09:00 ..
-rw-r--r--. 1 root root    16908 Dec  8 09:00 <a-bunch-of-these>.xml.zck
drwxr-xr-x. 1 root root        0 Dec  8 09:44 gen
-rw-r--r--. 1 root root     8306 Dec  8 09:00 repomd.xml
```

Now if I rerun `dnf --cacheonly group list --installed` (without sudo) it too succeeds.

So it looks like the 'gen' subdirectory being missing breaks `dnf --cacheonly` (when run without sudo) because the permissions on the parent directory don't allow regular users to create it. If it already exists, everything works.

Comment 1 Jonathan Watt 2021-12-08 10:28:50 UTC
Looking at the output from `sudo ls /var/cache/dnf/updates-*/repodata/` it seems that only one of the caches contains a 'gen' subdirectory. If that indicates that the 'gen' subdirectory is sometimes removed, that would also be an issue.

Comment 2 Marek Blaha 2021-12-16 11:42:31 UTC
Hi, the `gen` subdirectory is used by dnf to store decompressed versions of group metadata. E.g. repodata/e9328e1b5448d9e9f9b4908632a146aa768fd07ef2bc52c176ad85af06233170-comps-Everything.x86_64.xml.gz decompress into repodata/gen/groups.xml. The `gen` subdirectory does not get removed, but it is only part of repositories with comps groups metadata. Unfortunately the groups metadata are not decompressed when they are downloaded (thus as a root) but only once they are needed (e.g. by `dnf group` command which might by executed as regular user). 

Until this is fixed, can you as a workaround omit the `--cacheonly` parameter?

Comment 3 Jonathan Watt 2021-12-16 13:30:05 UTC
Thanks for your reply and for the explanation, Marek. For now we have indeed dropped the `--cacheonly` parameter. It would be nice if we could keep using it but it's a relatively small inconvenience.

Comment 4 Marek Blaha 2021-12-17 16:04:40 UTC
PR which should fix the issue: https://github.com/rpm-software-management/dnf/pull/1803

Comment 5 Fedora Update System 2022-03-14 13:07:17 UTC
FEDORA-2022-e119fcc7d5 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-e119fcc7d5

Comment 6 Fedora Update System 2022-03-14 13:09:44 UTC
FEDORA-2022-e119fcc7d5 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 7 Marek Blaha 2022-04-22 07:32:55 UTC
PR with test: https://github.com/rpm-software-management/ci-dnf-stack/pull/1087