Bug 2263968

Thanks. We've been waiving toolbox tests for quite sometime now. We're all set on this one. Closing..

(In reply to Lokesh Mandvekar from comment #1)
> We've been waiving toolbox tests for quite sometime now. We're all
> set on this one.

Oh, this is the first time I am hearing about this.

It seems like we are not even running the Toolbx tests:
https://artifacts.dev.testing-farm.io/aef9cb3b-7936-4a99-b1be-5cbbd8213ab0/work-tests.ymlg4id4qba/tests-b1w6pbo0/test.toolbox.bats.log

The reason is right there:

# test suite: Set up
# Missing dependencies
# Forgot to run 'git submodule init' and 'git submodule update' ?

I remember we spent a lot of effort on the tests, so we really should be running them.  Reopening.

I think I understand what's going on.

This was triggered by this:
"test/system: Clarify the use of Git submodules"
https://github.com/containers/toolbox/commit/2c0960660330dc6b
https://github.com/containers/toolbox/pull/1298

The upstream Toolbx CI runs the tests with 'bats ./test/system', while the downstream Podman CI runs them as 'bats .'.  That's confusing the paths.

I am not fully sure, but I think we need to parametrize it with $BATS_TEST_DIRNAME:
https://bats-core.readthedocs.io/en/stable/writing-tests.html

This might work:
https://github.com/containers/toolbox/pull/1448

I can include it in Fedora's toolbox package.  Is there a good way to trigger Podman's downstream Fedora CI?  Or will we have to wait for the next podman build?

I have now submitted the toolbox updates:
https://bodhi.fedoraproject.org/updates/FEDORA-2024-308ed7fb4e
https://bodhi.fedoraproject.org/updates/FEDORA-2024-7f2ce8d995
https://bodhi.fedoraproject.org/updates/FEDORA-2024-729ce9f37e
https://bodhi.fedoraproject.org/updates/FEDORA-2024-b7df508910

How can we check that this actually fixes Podman's Fedora CI?

You can look at the "fedora-ci.koji-build.tier0.functional" results in the "Automated Tests" tab of the Bodhi update. It unfortunately looks like they're still failing. If you prefer, you can use the CI on distgit PRs or set up a local environment with tmt or sti (depending on which framework the package uses) to run the tests without submitting an entire build.

(In reply to Maxwell G from comment #7)
> You can look at the "fedora-ci.koji-build.tier0.functional" results in the
> "Automated Tests" tab of the Bodhi update. It unfortunately looks like
> they're still failing.

Which Bodhi update?  I guess the tests on the Toolbx Bodhi updates are not the same as the one on Podman's Bodhi updates, right?  I think those are failing for some other, potentially unrelated, reason.

I hope we have made some progress beyond the failure that I found in comment 2

> If you prefer, you can use the CI on distgit PRs or
> set up a local environment with tmt or sti (depending on which framework the
> package uses) to run the tests without submitting an entire build.

Part of the problem is that I don't know very well how the downstream Fedora CI works.  :)

(In reply to Debarshi Ray from comment #6)
> I have now submitted the toolbox updates:
> https://bodhi.fedoraproject.org/updates/FEDORA-2024-308ed7fb4e
> https://bodhi.fedoraproject.org/updates/FEDORA-2024-7f2ce8d995
> https://bodhi.fedoraproject.org/updates/FEDORA-2024-729ce9f37e
> https://bodhi.fedoraproject.org/updates/FEDORA-2024-b7df508910
> 
> How can we check that this actually fixes Podman's Fedora CI?

These led to:

1..306
# test suite: Set up
# test suite: Tear down
not ok 1 setup_suite
# (from function `setup_suite' in test file ./setup_suite.bash, line
    55)
#   `_pull_and_cache_distro_image fedora "$((system_version-1))" ||
    false' failed
# Failed to cache image registry.fedoraproject.org/fedora-toolbox:40
    to /tmp/bats-run-IPz4Cn/image-cache/fedora-toolbox-40
# time="2024-02-19T11:41:43Z" level=fatal msg="copying system image
    from manifest list: writing blob: write
    /tmp/bats-run-IPz4Cn/image-cache/fedora-toolbox-40/dir-put-blob607392514:
    no space left on device"
# bats warning: Executed 1 instead of expected 306 tests

... which were fixed by:
https://src.fedoraproject.org/rpms/podman/pull-request/128

Now we get some failures like this one:

not ok 110 run: Smoke test with true(1)
# (from function `assert_output' in file ./libs/bats-assert/src/assert.bash, line 255,
#  in test file ./104-run.bats, line 38)
#   `assert_output ""' failed
#
# -- output differs --
# expected (0 lines):
#
# actual (3 lines):
#   Error: crun: chdir to `/usr/share/toolbox/test/system`: No such file or directory: OCI runtime attempted to invoke a command that was not found
#   Error: directory /usr/share/toolbox/test/system not found in container fedora-toolbox-41
#   Using /home/testuser instead.
# --
#

(In reply to Debarshi Ray from comment #9)
> Now we get some failures like this one:
> 
> not ok 110 run: Smoke test with true(1)
> # (from function `assert_output' in file ./libs/bats-assert/src/assert.bash,
> line 255,
> #  in test file ./104-run.bats, line 38)
> #   `assert_output ""' failed
> #
> # -- output differs --
> # expected (0 lines):
> #
> # actual (3 lines):
> #   Error: crun: chdir to `/usr/share/toolbox/test/system`: No such file or
> directory: OCI runtime attempted to invoke a command that was not found
> #   Error: directory /usr/share/toolbox/test/system not found in container
> fedora-toolbox-41
> #   Using /home/testuser instead.
> # --
> #

This might work:
https://github.com/containers/toolbox/pull/1457

I have now submitted the toolbox updates:
https://bodhi.fedoraproject.org/updates/FEDORA-2024-72ab5458c8
https://bodhi.fedoraproject.org/updates/FEDORA-2024-154c8d783e
https://bodhi.fedoraproject.org/updates/FEDORA-2024-369385b03e
https://bodhi.fedoraproject.org/updates/FEDORA-2024-762d042d4e

This bug appears to have been reported against 'rawhide' during the Fedora Linux 42 development cycle.
Changing version to 42.

Closing. BTW, we now have podman and toolbox tests triggered via TMT on Fedora and there have been fewer flakes and failures recently.