Bug 2024620

Summary:	Rawhide composes failed with mock-2.14-1.fc34
Product:	[Fedora] Fedora	Reporter:	Mohan Boddu <mboddu>
Component:	mock	Assignee:	Copr Team <copr-team>
Status:	CLOSED ERRATA	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	34	CC:	awilliam, copr-team, jkeating, mebrown, msuchy, philip.wyett, praiskup, williams
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	mock-2.15-1.fc35 mock-2.15-1.el8 mock-2.15-1.fc34 mock-2.15-1.el7 mock-2.15-1.fc33	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-11-24 01:19:14 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Mohan Boddu 2021-11-18 14:07:58 UTC

Description of problem:
Yesterday all the builders got updated to mock-2.14-1.fc34 and it failed the runroot tasks in rawhide compose

Task id: https://koji.fedoraproject.org/koji/taskinfo?taskID=79008990

```
Traceback (most recent call last):
  File "/usr/libexec/mock/mock", line 1078, in <module>
    exitStatus = main()
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 841, in main
    result = run_command(options, args, config_opts, commands, buildroot, state)
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 897, in run_command
    return commands.chroot(args, options)
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.9/site-packages/mockbuild/backend.py", line 405, in chroot
    result = self.buildroot.doChroot(args, shell=shell, cwd=options.cwd,
  File "/usr/lib/python3.9/site-packages/mockbuild/buildroot.py", line 280, in doChroot
    result = util.do_with_status(command, chrootPath=self.make_chroot_path(),
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.9/site-packages/mockbuild/util.py", line 544, in do_with_status
    child = subprocess.Popen(
  File "/usr/lib64/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib64/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '--'
```

Version-Release number of selected component (if applicable):
mock-2.14-1.fc34

Comment 1 Pavel Raiskup 2021-11-18 14:11:57 UTC

Thank you for the report, Mohan.

This is probably related to the optparse => argparse rewrite.  argparse handles
the `--` separator differently, and there must be some bug.  Can you please tell
what is the exact mock command being executed?

Comment 2 Mohan Boddu 2021-11-18 17:10:22 UTC

I think koji runs mock as `mock --chroot -- {command_list}`, the `--` separator might be the problem here.

Here's the command:

```
Running in chroot: ['--', '/bin/sh', '-c', '{ rm -f /var/lib/rpm/__db*; rm -rf /var/cache/yum/*; set -x; rm -rf /mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/buildinstall/Server && lorax --product=Fedora --version=Rawhide --release=Rawhide --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-swig-4.0-3620210916145835-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-IO-stringy-2.113-3620210920105958-2 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-postgresql-11-3620210824092915-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-postgresql-10-3620211019194202-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-postgresql-12-3620210816122311-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-libwww-perl-6.48-3620210903113609-81d42e4a --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-XML-Parser-2.46-3620210907103304-1 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-IO-Socket-SSL-2.068-3620210903104856-7b91f26c --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-postgresql-14-3620211013112805-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-YAML-1.30-3620210906131152-1 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-FCGI-0.79-3620210907081151-1 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-Date-Manip-6.83-3620210903123042-1 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-DBI-1.643-3620210903124335-1 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-DBD-Pg-3.14-3620210906120916-1 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-CGI-4.51-3620211004091221-1 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-DBD-SQLite-1.66-3620210906121309-1 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-App-cpanminus-1.7044-3620210907100605-1 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-openmpi-4.0-3620211005013734-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-DBD-MySQL-4.050-3620210906094406-1 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-nodejs-16-3620211108153954-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-nodejs-14-3620211014121918-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-nginx-mainline-3620211102232012-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-nginx-1.20-3620210908185433-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-dwm-latest-3620201101015024-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-mysql-8.0-3620211031142409-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-dwm-6.2-3620210202111826-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-dwm-6.1-3620210201221442-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-dwm-6.0-3620210201213909-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-cri-o-1.22-3620211112151636-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-cri-o-1.21-3620210818145024-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-avocado-vt-latest-3620210902083930-f4f95988 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-avocado-vt-82lts-3620210902113311-a47ea332 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-5.32-3620210922083143-553f8e7a --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-ghc-9.2-3620211101111632-d099bf28 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-ghc-8.10-3620210920031153-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/f36 --source=https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/comps_repo_Server --variant=Server --noupgrade --buildarch=x86_64 --volid=Fedora-S-dvd-x86_64-rawh --logfile=/mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/logs/x86_64/buildinstall-Server-logs/lorax.log --rootfs-size=3 --squashfs-only /mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/buildinstall/Server ; EXIT_CODE=$? ; chmod -R a+r /mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/buildinstall/Server /mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/logs/x86_64/buildinstall-Server-logs ; chown -R 0 /mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/buildinstall/Server /mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/logs/x86_64/buildinstall-Server-logs ; exit $EXIT_CODE; } < /dev/null 2>&1 | /usr/bin/tee /builddir/runroot.log; exit ${PIPESTATUS[0]}']
```

Comment 3 Adam Williamson 2021-11-18 17:13:05 UTC

So what's happening here is:

1. pungi calls `koji runroot` - see https://pagure.io/pungi/blob/master/f/pungi/wrappers/kojiwrapper.py#_105 and https://pagure.io/pungi/blob/master/f/pungi/wrappers/kojiwrapper.py#_88
2. `koji runroot` calls `mock --chroot -- (commands)` - see https://pagure.io/koji/blob/master/f/plugins/builder/runroot.py#_223

That `--` is documented as allowable in the mock manpage:

       --chroot [--] COMMAND [ARGS...]
              Run the specified command non-interactively within the chroot (no --clean is performed).

but I think with the argparse port we are not properly ignoring it. The argparse port did handle this issue at the level of the main parser - so `mock -- something` would be dealt with - but I think not at the level of the subparser.

Comment 4 Adam Williamson 2021-11-18 17:25:52 UTC

well, hm, now I grok it a bit harder, the code in mock that tries to deal with this sort of *ought* to deal with it in that case, because `args` should contain all arguments that are unknown to the parser, and `--` should be the first of those (as `--chroot` *is* known to the parser). So there may be more to this. There may be *another* unknown argument in the command koji constructs, somehow. Let me look a bit harder.

Comment 5 Adam Williamson 2021-11-18 17:52:38 UTC

Hum. So, yeah, I still can't quite see why this is happening. But a couple of notes:

1. I can reproduce the problem by adding *another* `--`, as we only handle *one*.
2. We only actually error out here with `--old-chroot` (that is, `--isolation=simple`). With nspawn isolation, even if we get a bare `--` into the command list, mock exits successfully. That is:
* `mock -r fedora-rawhide-x86_64 --chroot -- -- ls` shows INFO: Running in chroot: ['--', 'ls'], but doesn't error out - we see the output of the `ls` command, the exit status is 0, and there's no apparent error message from the attempt to run `--`
* `mock -r fedora-rawhide-x86_64 --old-chroot --chroot -- -- ls` shows INFO: Running in chroot: ['--', 'ls'] , then crashes with `FileNotFoundError: [Errno 2] No such file or directory: '--'`

I don't see where an extra `--` would be coming in to trigger this, though. My attempt to reproduce what koji actually does is this:

#!/bin/python3

import os

cmd = ["/usr/bin/mock", "-r", "fedora-rawhide-x86_64", "--arch", "x86_64", "--old-chroot", "--chroot", "--", "ls"]
os.execvp(cmd[0], cmd)

but that doesn't fail, it works OK. I cannot see anywhere we'd be getting an extra "--" from, either.

Comment 6 Adam Williamson 2021-11-18 18:04:34 UTC

Aha. I think https://pagure.io/koji/pull-request/3130 would actually fix this, and explains why I couldn't see why it was failing (as I was looking at the *current* koji code, i.e. *with that PR merged*).

Previous to that PR, koji was doing this:

["/usr/bin/mock", "-r", "fedora-rawhide-x86_64", "--arch", "x86_64", "--old-chroot", "chroot", "--", "ls"]

and that effectively sandbags the code that tries to handle the `--`, because `args[0]` is now "chroot", not "--". This should be fixable in mock by swapping "# handle old-style commands" to happen before "# Optparse.parse_args() eats '--' argument, while argparse doesn't.  Do it manually."

Comment 7 Adam Williamson 2021-11-18 18:14:42 UTC

so deploying *either one or both* of the koji and mock changes on the builders should resolve this.

Comment 8 Fedora Update System 2021-11-18 20:48:25 UTC

FEDORA-EPEL-2021-d5b39e1cc0 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-d5b39e1cc0

Comment 9 Fedora Update System 2021-11-18 20:48:26 UTC

FEDORA-2021-2ca5dcfa9e has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-2ca5dcfa9e

Comment 10 Fedora Update System 2021-11-18 20:48:27 UTC

FEDORA-2021-1c063ee673 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-1c063ee673

Comment 11 Fedora Update System 2021-11-19 01:51:58 UTC

FEDORA-2021-4f0e4d759c has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-4f0e4d759c`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-4f0e4d759c

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 12 Fedora Update System 2021-11-19 02:24:35 UTC

FEDORA-EPEL-2021-d5b39e1cc0 has been pushed to the Fedora EPEL 7 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-d5b39e1cc0

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 13 Fedora Update System 2021-11-19 02:40:43 UTC

FEDORA-2021-2ca5dcfa9e has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-2ca5dcfa9e`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-2ca5dcfa9e

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 14 Fedora Update System 2021-11-19 18:24:04 UTC

FEDORA-EPEL-2021-5f4f9c022a has been pushed to the Fedora EPEL 8 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-5f4f9c022a

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 15 Fedora Update System 2021-11-19 19:09:42 UTC

FEDORA-2021-1c063ee673 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-1c063ee673`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-1c063ee673

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 16 Fedora Update System 2021-11-24 01:19:14 UTC

FEDORA-2021-1c063ee673 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 17 Fedora Update System 2021-11-27 00:33:38 UTC

FEDORA-EPEL-2021-5f4f9c022a has been pushed to the Fedora EPEL 8 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 18 Fedora Update System 2021-11-27 01:16:34 UTC

FEDORA-2021-4f0e4d759c has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 19 Fedora Update System 2021-11-27 01:31:08 UTC

FEDORA-EPEL-2021-d5b39e1cc0 has been pushed to the Fedora EPEL 7 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 20 Fedora Update System 2021-11-27 01:51:42 UTC

FEDORA-2021-2ca5dcfa9e has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.