Description of problem: podman was rebuilt yesterday as part of a mass rebuild of Go packages for the golang 1.22.0 update. The package's gating tests are failing[1] which is preventing pushing the update. Can you please look at the failure and either confirm that it's a false positive so I can waive it or submit a new build with fixed tests? [1] https://bodhi.fedoraproject.org/updates/FEDORA-2024-256b176c60 Version-Release number of selected component (if applicable): podman-5.0.0~rc1-3.fc40
Thanks. We've been waiving toolbox tests for quite sometime now. We're all set on this one. Closing..
(In reply to Lokesh Mandvekar from comment #1) > We've been waiving toolbox tests for quite sometime now. We're all > set on this one. Oh, this is the first time I am hearing about this. It seems like we are not even running the Toolbx tests: https://artifacts.dev.testing-farm.io/aef9cb3b-7936-4a99-b1be-5cbbd8213ab0/work-tests.ymlg4id4qba/tests-b1w6pbo0/test.toolbox.bats.log The reason is right there: # test suite: Set up # Missing dependencies # Forgot to run 'git submodule init' and 'git submodule update' ? I remember we spent a lot of effort on the tests, so we really should be running them. Reopening.
I think I understand what's going on. This was triggered by this: "test/system: Clarify the use of Git submodules" https://github.com/containers/toolbox/commit/2c0960660330dc6b https://github.com/containers/toolbox/pull/1298 The upstream Toolbx CI runs the tests with 'bats ./test/system', while the downstream Podman CI runs them as 'bats .'. That's confusing the paths.
I am not fully sure, but I think we need to parametrize it with $BATS_TEST_DIRNAME: https://bats-core.readthedocs.io/en/stable/writing-tests.html
This might work: https://github.com/containers/toolbox/pull/1448 I can include it in Fedora's toolbox package. Is there a good way to trigger Podman's downstream Fedora CI? Or will we have to wait for the next podman build?
I have now submitted the toolbox updates: https://bodhi.fedoraproject.org/updates/FEDORA-2024-308ed7fb4e https://bodhi.fedoraproject.org/updates/FEDORA-2024-7f2ce8d995 https://bodhi.fedoraproject.org/updates/FEDORA-2024-729ce9f37e https://bodhi.fedoraproject.org/updates/FEDORA-2024-b7df508910 How can we check that this actually fixes Podman's Fedora CI?
You can look at the "fedora-ci.koji-build.tier0.functional" results in the "Automated Tests" tab of the Bodhi update. It unfortunately looks like they're still failing. If you prefer, you can use the CI on distgit PRs or set up a local environment with tmt or sti (depending on which framework the package uses) to run the tests without submitting an entire build.
(In reply to Maxwell G from comment #7) > You can look at the "fedora-ci.koji-build.tier0.functional" results in the > "Automated Tests" tab of the Bodhi update. It unfortunately looks like > they're still failing. Which Bodhi update? I guess the tests on the Toolbx Bodhi updates are not the same as the one on Podman's Bodhi updates, right? I think those are failing for some other, potentially unrelated, reason. I hope we have made some progress beyond the failure that I found in comment 2 > If you prefer, you can use the CI on distgit PRs or > set up a local environment with tmt or sti (depending on which framework the > package uses) to run the tests without submitting an entire build. Part of the problem is that I don't know very well how the downstream Fedora CI works. :)
(In reply to Debarshi Ray from comment #6) > I have now submitted the toolbox updates: > https://bodhi.fedoraproject.org/updates/FEDORA-2024-308ed7fb4e > https://bodhi.fedoraproject.org/updates/FEDORA-2024-7f2ce8d995 > https://bodhi.fedoraproject.org/updates/FEDORA-2024-729ce9f37e > https://bodhi.fedoraproject.org/updates/FEDORA-2024-b7df508910 > > How can we check that this actually fixes Podman's Fedora CI? These led to: 1..306 # test suite: Set up # test suite: Tear down not ok 1 setup_suite # (from function `setup_suite' in test file ./setup_suite.bash, line 55) # `_pull_and_cache_distro_image fedora "$((system_version-1))" || false' failed # Failed to cache image registry.fedoraproject.org/fedora-toolbox:40 to /tmp/bats-run-IPz4Cn/image-cache/fedora-toolbox-40 # time="2024-02-19T11:41:43Z" level=fatal msg="copying system image from manifest list: writing blob: write /tmp/bats-run-IPz4Cn/image-cache/fedora-toolbox-40/dir-put-blob607392514: no space left on device" # bats warning: Executed 1 instead of expected 306 tests ... which were fixed by: https://src.fedoraproject.org/rpms/podman/pull-request/128 Now we get some failures like this one: not ok 110 run: Smoke test with true(1) # (from function `assert_output' in file ./libs/bats-assert/src/assert.bash, line 255, # in test file ./104-run.bats, line 38) # `assert_output ""' failed # # -- output differs -- # expected (0 lines): # # actual (3 lines): # Error: crun: chdir to `/usr/share/toolbox/test/system`: No such file or directory: OCI runtime attempted to invoke a command that was not found # Error: directory /usr/share/toolbox/test/system not found in container fedora-toolbox-41 # Using /home/testuser instead. # -- #
(In reply to Debarshi Ray from comment #9) > Now we get some failures like this one: > > not ok 110 run: Smoke test with true(1) > # (from function `assert_output' in file ./libs/bats-assert/src/assert.bash, > line 255, > # in test file ./104-run.bats, line 38) > # `assert_output ""' failed > # > # -- output differs -- > # expected (0 lines): > # > # actual (3 lines): > # Error: crun: chdir to `/usr/share/toolbox/test/system`: No such file or > directory: OCI runtime attempted to invoke a command that was not found > # Error: directory /usr/share/toolbox/test/system not found in container > fedora-toolbox-41 > # Using /home/testuser instead. > # -- > # This might work: https://github.com/containers/toolbox/pull/1457
I have now submitted the toolbox updates: https://bodhi.fedoraproject.org/updates/FEDORA-2024-72ab5458c8 https://bodhi.fedoraproject.org/updates/FEDORA-2024-154c8d783e https://bodhi.fedoraproject.org/updates/FEDORA-2024-369385b03e https://bodhi.fedoraproject.org/updates/FEDORA-2024-762d042d4e
This bug appears to have been reported against 'rawhide' during the Fedora Linux 42 development cycle. Changing version to 42.
Closing. BTW, we now have podman and toolbox tests triggered via TMT on Fedora and there have been fewer flakes and failures recently.