It seems that some bodhi-server tests that try to run dnf in a subprocess are failing when are run in Koji due to 'Permission denied' failures. For example: E subprocess.CalledProcessError: Command '['dnf', '--disablerepo=*', '--repofrompath=testrepo,/tmp/tmpyvrt9jwwbodhi/', '--enablerepo=testrepo', '--setopt=skip_if_unavailable=0', '--setopt=testrepo.skip_if_unavailable=0', '--refresh', '--nogpgcheck', 'list', 'available']' returned non-zero exit status 1. /usr/lib64/python3.11/subprocess.py:571: CalledProcessError ----------------------------- Captured stderr call ----------------------------- Failed to store expired repos cache: [Errno 13] Permission denied: '/var/cache/yum/expired_repos.json' Error: Cannot create repo destination directory "/var/cache/yum/testrepo-b2096c9726601dda": Permission denied https://koji.fedoraproject.org/koji/taskinfo?taskID=96058001 I cannot reproduce those failures when running tests directly or when I try a rebuild with mock or in COPR.
Could it somehow be related to running on a ppc64le instance? you can use 'ppc64le-test.fedorainfracloud.org' to try it in mock. Does it fail the same way there?
I cannot find 'ppc64le-test.fedorainfracloud.org' in mock configurations, is it provided by some specific package other than mock itself? However, I fired a build on COPR on all arches, ppc64le included, and everything went fine: https://copr.fedorainfracloud.org/coprs/mattia/bodhi-test/build/5228330/ The latest successfully build on Koji was on 29/11: https://koji.fedoraproject.org/koji/buildinfo?buildID=2094544 and I see that it started to fail in: https://koji.fedoraproject.org/koji/buildinfo?buildID=2107719 I see that the first was using kernel version == 5.19.9-200.fc36.s390x while the failing one kernel version == 6.0.15-300.fc37.ppc64le there is a thread on devel mailing list about kernel 6.0.x breaking some python tests: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/7VPNMC77YC3SI5LFYKUA4B5MTFPLTLVB/ all COPR builds have been run on kernel <= 6.0.8, while, for example, this scratch build was run on kernel version == 6.0.10-300.fc37.aarch64 : https://koji.fedoraproject.org/koji/taskinfo?taskID=96098545 Can that be the cause?
(In reply to Mattia Verga from comment #2) > I cannot find 'ppc64le-test.fedorainfracloud.org' in mock configurations, is > it provided by some specific package other than mock itself? No, it's a machine. ;) You ssh into it with your fedora key and can run mock there. ;) But It seems unlikely it's a ppc problem. > However, I fired a build on COPR on all arches, ppc64le included, and > everything went fine: > https://copr.fedorainfracloud.org/coprs/mattia/bodhi-test/build/5228330/ > > The latest successfully build on Koji was on 29/11: > https://koji.fedoraproject.org/koji/buildinfo?buildID=2094544 > > and I see that it started to fail in: > https://koji.fedoraproject.org/koji/buildinfo?buildID=2107719 > > I see that the first was using > kernel version == 5.19.9-200.fc36.s390x > while the failing one > kernel version == 6.0.15-300.fc37.ppc64le > > there is a thread on devel mailing list about kernel 6.0.x breaking some > python tests: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/ > thread/7VPNMC77YC3SI5LFYKUA4B5MTFPLTLVB/ > all COPR builds have been run on kernel <= 6.0.8, while, for example, this > scratch build was run on kernel version == 6.0.10-300.fc37.aarch64 : > https://koji.fedoraproject.org/koji/taskinfo?taskID=96098545 > > Can that be the cause? I wouldn't think so, but possibly? In any case I have updated all the builders to 6.1.5 kernel now and cleaned up a few that had issues (including the ppc64le one that failed the build above). I also did a scratch build of bodhi-server and it completed fine: https://koji.fedoraproject.org/koji/taskinfo?taskID=96135787 can you check it now?
That scratch build completed fine because I currently excluded those tests that requires to run a 'dnf' command. I've fired another scratch build with the test re-enabled and they're still failing: https://koji.fedoraproject.org/koji/taskinfo?taskID=96152155
Ah, ok. I'll try and take a look. The one thing that stands out is that it's trying to use /var/cache/yum... which... no package provides anymore. It should be /var/cache/dnf/ But I would expect that to file in local mock just as much as koji mock. ;(
Looking at the builder: # ls -la /var/lib/mock/f38-build-40315738-4974586/root/var/cache/yum total 74816 drwxrwxr-x. 1 root root 142 Jan 15 08:06 . drwxr-xr-x. 1 root root 34 Jan 15 08:01 .. drwxr-xr-x. 1 root root 32 Jan 15 08:00 build-3c6a7d377cffff40 -rw-r--r--. 1 root root 49987208 Jan 15 08:00 build-filenames.solvx -rw-r--r--. 1 root root 26511322 Jan 15 08:00 build.solv -rw-r--r--. 1 root root 2 Jan 15 08:06 expired_repos.json so, something is making those as root and when the build user tries to do something with them it can't. ;( But what? ;(
Indeed, when running the build in mock, the buildroot uses /var/cache/dnf: $ ls -la totale 128708 drwxr-xr-x. 1 root root 544 16 gen 18.08 . drwxr-xr-x. 1 root root 34 16 gen 18.08 .. -rw-r--r--. 1 root root 21 16 gen 18.08 expired_repos.json drwxr-xr-x. 1 root root 56 7 gen 10.44 fedora-a3256cee7c7d69ad -rw-r--r--. 1 root root 55801422 7 gen 10.44 fedora-filenames.solvx -rw-r--r--. 1 root root 26546523 13 gen 19.07 fedora.solv drwxr-xr-x. 1 root root 56 16 gen 18.05 updates-fd4d3d0d1c34d49a -rw-r--r--. 1 root root 19676260 16 gen 18.06 updates-filenames.solvx -rw-r--r--. 1 root root 6110579 16 gen 18.06 updates.solv drwxr-xr-x. 1 root root 56 16 gen 18.08 updates-testing-77f7a183e9a6bed9 -rw-r--r--. 1 root root 6835785 16 gen 18.08 updates-testing-filenames.solvx -rw-r--r--. 1 root root 3102925 16 gen 18.08 updates-testing.solv -rw-r--r--. 1 root root 5018150 16 gen 18.08 updates-testing-updateinfo.solvx -rw-r--r--. 1 root root 8682410 16 gen 18.06 updates-updateinfo.solvx However root ownership seems right, though.
This bug appears to have been reported against 'rawhide' during the Fedora Linux 38 development cycle. Changing version to 38.
This have been fixed somehow... the tests are now passing again.