@lhrazky, @jblazek PTAL (once got a chance)

Thank you! Jan

JFTR, tried with the latest "db39bd4846dc" from RHBA-2020:0354. The behaviour above happens when building with docker (docker-1.13.1-104.git4ef4b30.el7.x86_64) on Red Hat Enterprise Linux 7 system (redhat-release-server-7.7-10.el7.x86_64). Didn't try with podman yet.

Hey Jan,

judging by this line in your microdnf output:

(process:13): libdnf-WARNING **: 12:09:07.031: Loading "/etc/dnf/dnf.conf": IniParser: Can't open file

There's some problem with your dnf.conf, possibly a syntax error? Once you find out what's wrong with it, please let me know.

Sorry about the abort. I'll have a look, not sure if we're gonna be able to handle it better... let me just leave it at stating the API is far from ideal in this regard.

Ok, so that "IniParser: Can't open file" error is actually not relevant, although it's probably something you wanna fix (I get it when /etc/dnf/dnf.conf doesn't exist)

The problem is with '/var/cache/yum/metadata/rhel-8-for-x86_64-appstream-rpms-8-x86_64/repodata/5516bc5ae7bee90ae413e4bc9e2b2a0b889b6a416256f50d5cecf16e31956e9e-modules.yaml.gz', I've missed that...

Can you check the file? There's likely a problem with it. You didn't mention what you're testing this on at all, so I can't try to reproduce.

Also, yes, this is totally unrelated to the bug you've cloned, so that also just brings additional noise into the report...

The same issue happens with podman (podman-1.6.4-2.module+el8.1.1+5363+bf8ff1af.x86_64) on Red Hat Enterprise Linux 8 (redhat-release-8.0-0.44.el8.x86_64):

2020-02-04 14:26:12,191 cekit        DEBUG    Running Podman build: '/usr/bin/podman build --squash -t redhat-sso-7/sso74-openshift-rhel8:7 -t redhat-sso-7/sso74-openshift-rhel8:latest target/image'
STEP 1: FROM ubi8-minimal:latest
STEP 2: USER root
STEP 3: COPY repos/content_sets_odcs.repo     /etc/yum.repos.d/
STEP 4: COPY modules/bz-1769831 /tmp/scripts/bz-1769831
STEP 5: USER root
STEP 6: RUN [ "sh", "-x", "/tmp/scripts/bz-1769831/install" ]
++ id -u
+ mkdir -p /run/user/0
STEP 7: COPY modules/jboss.container.user /tmp/scripts/jboss.container.user
STEP 8: USER root
STEP 9: RUN microdnf --setopt=tsflags=nodocs install -y unzip tar rsync shadow-utils             && microdnf clean all             && rpm -q unzip tar rsync shadow-utils

(process:6): libdnf-WARNING **: 19:26:14.876: Loading "/etc/dnf/dnf.conf": IniParser: Can't open file
Downloading metadata...
Downloading metadata...
Downloading metadata...
Downloading metadata...
Downloading metadata...
Downloading metadata...
Downloading metadata...
terminate called after throwing an instance of 'libdnf::File::NotOpenedException'
  what():  File '/var/cache/yum/metadata/rhel-8-for-x86_64-appstream-rpms-8-x86_64/repodata/b2ac940675f8f17130db983ab4b49e92a890d72db11ca3eeca8b9196354446f1-modules.yaml.gz' is not opened.
/bin/sh: line 1:     6 Aborted                 (core dumped) microdnf --setopt=tsflags=nodocs install -y unzip tar rsync shadow-utils
Error: error building at STEP "RUN microdnf --setopt=tsflags=nodocs install -y unzip tar rsync shadow-utils             && microdnf clean all             && rpm -q unzip tar rsync shadow-utils": error while running runtime: exit status 134
2020-02-04 14:28:24,726 cekit        ERROR    Image build failed, see logs above.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/cekit/builders/podman.py", line 55, in run
    subprocess.check_call(cmd)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/podman', 'build', '--squash', '-t', 'redhat-sso-7/sso74-openshift-rhel8:7', '-t', 'redhat-sso-7/sso74-openshift-rhel8:latest', 'target/image']' returned non-zero exit status 125.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/cekit/cli.py", line 335, in run
    command.execute()
  File "/usr/local/lib/python3.6/site-packages/cekit/builder.py", line 67, in execute
    self.run()
  File "/usr/local/lib/python3.6/site-packages/cekit/builders/podman.py", line 59, in run
    raise CekitError("Image build failed, see logs above.")
cekit.errors.CekitError: Image build failed, see logs above.

(In reply to Lukáš Hrázký from comment #4)
> Ok, so that "IniParser: Can't open file" error is actually not relevant,
> although it's probably something you wanna fix (I get it when
> /etc/dnf/dnf.conf doesn't exist)

The image in question (that is attempted to be built) is using content sets:
     https://github.com/jboss-container-images/redhat-sso-7-openshift-image/blob/sso74-dev/image.yaml#L128

so that warning about absent "/etc/dnf/dnf.conf" shouldn't matter AFAICT (similar one was present on Red Hat Enterprise Linux 7, when building with yum, and the image build was successfull). So AFAICT the warning is benign.


> 
> The problem is with
> '/var/cache/yum/metadata/rhel-8-for-x86_64-appstream-rpms-8-x86_64/repodata/
> 5516bc5ae7bee90ae413e4bc9e2b2a0b889b6a416256f50d5cecf16e31956e9e-modules.
> yaml.gz', I've missed that...
> 
> Can you check the file? There's likely a problem with it. You didn't mention
> what you're testing this on at all, so I can't try to reproduce.

That file doesn't seem to exist at all:

# docker run --rm -ti 18518250279c bash
[root@8a8e9f3fac7c /]# microdnf --setopt=tsflags=nodocs install -y unzip tar rsync shadow-utils             && microdnf clean all             && rpm -q unzip tar rsync shadow-utils

(process:10): libdnf-WARNING **: 19:38:14.003: Loading "/etc/dnf/dnf.conf": IniParser: Can't open file
Downloading metadata...
Downloading metadata...
Downloading metadata...
Downloading metadata...
Downloading metadata...
Downloading metadata...
Downloading metadata...
terminate called after throwing an instance of 'libdnf::File::NotOpenedException'
  what():  File '/var/cache/yum/metadata/rhel-8-for-x86_64-appstream-rpms-8-x86_64/repodata/eabc49edbe7a2e3f2f8466e80f05ed2e4968821eb854117888028a9acd66d323-modules.yaml.gz' is not opened.
Aborted (core dumped)

[root@8a8e9f3fac7c /]# stat -c %a /var/cache/yum/metadata/rhel-8-for-x86_64-appstream-rpms-8-x86_64/repodata/eabc49edbe7a2e3f2f8466e80f05ed2e4968821eb854117888028a9acd66d323-modules.yaml.gz
stat: cannot stat '/var/cache/yum/metadata/rhel-8-for-x86_64-appstream-rpms-8-x86_64/repodata/eabc49edbe7a2e3f2f8466e80f05ed2e4968821eb854117888028a9acd66d323-modules.yaml.gz': No such file or directory

> 
> Also, yes, this is totally unrelated to the bug you've cloned, so that also
> just brings additional noise into the report...

I'm still not able to reproduce, you did not provide a complete reproducer. You have this `repos/content_sets_odcs.repo` file you're copying into `/etc/yum.repos.d/` which is probably setting up the repository on which it's failing later.

I don't see the repository URL so I can't check that either. But I am quite certain the repository metadata are inconsistent. If microdnf is trying to open that `*-modules.yaml.gz` file, it means it is listed in `repomd.xml`, but actually missing from the repository. You can check that yourself or give me the configuration for the failing repository and I can check it for you.

Hello Lukáš,

(In reply to Lukáš Hrázký from comment #7)
> I'm still not able to reproduce, you did not provide a complete reproducer.
> You have this `repos/content_sets_odcs.repo` file you're copying into
> `/etc/yum.repos.d/` which is probably setting up the repository on which
> it's failing later.
> 
> I don't see the repository URL so I can't check that either. But I am quite
> certain the repository metadata are inconsistent. If microdnf is trying to
> open that `*-modules.yaml.gz` file, it means it is listed in `repomd.xml`,
> but actually missing from the repository. You can check that yourself or
> give me the configuration for the failing repository and I can check it for
> you.

  see the attached archive. It crashes for me consistently (each time), when
running on RHEL-7 via docker (docker-1.13.1-109.gitcccb291.el7_7.x86_64), and
running on RHEL-8 via podman (podman-1.6.4-2.module+el8.1.1+5363+bf8ff1af.x86_64).

Both of the systems are local VMs running via virt-manager (but both of them
can properly reach the Pulp repos in question, so the problem seems to be elsewhere).

Strange enough, it works (without crash) on Fedora 31 (physical machine) with both:
* docker (docker-1.13.1-68.git47e2230.fc30.x86_64), JFTR using the "systemd.unified_cgroup_hierarchy=0" workaround (should this matter)
* podman (podman-1.7.0-2.fc31.x86_64)

HTH

Thank you!
Jan

Ok. This error was almost certainly fixed already. I just checked the code and I can't see how the FileNotOpenedException error (which is an error in the program logic) could occur in current master. Marking fixed in CURRENTRELEASE (is that correct?).