Description of problem: Yesterday all the builders got updated to mock-2.14-1.fc34 and it failed the runroot tasks in rawhide compose Task id: https://koji.fedoraproject.org/koji/taskinfo?taskID=79008990 ``` Traceback (most recent call last): File "/usr/libexec/mock/mock", line 1078, in <module> exitStatus = main() File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace result = func(*args, **kw) File "/usr/libexec/mock/mock", line 841, in main result = run_command(options, args, config_opts, commands, buildroot, state) File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace result = func(*args, **kw) File "/usr/libexec/mock/mock", line 897, in run_command return commands.chroot(args, options) File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace result = func(*args, **kw) File "/usr/lib/python3.9/site-packages/mockbuild/backend.py", line 405, in chroot result = self.buildroot.doChroot(args, shell=shell, cwd=options.cwd, File "/usr/lib/python3.9/site-packages/mockbuild/buildroot.py", line 280, in doChroot result = util.do_with_status(command, chrootPath=self.make_chroot_path(), File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace result = func(*args, **kw) File "/usr/lib/python3.9/site-packages/mockbuild/util.py", line 544, in do_with_status child = subprocess.Popen( File "/usr/lib64/python3.9/subprocess.py", line 951, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib64/python3.9/subprocess.py", line 1821, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: '--' ``` Version-Release number of selected component (if applicable): mock-2.14-1.fc34
Thank you for the report, Mohan. This is probably related to the optparse => argparse rewrite. argparse handles the `--` separator differently, and there must be some bug. Can you please tell what is the exact mock command being executed?
I think koji runs mock as `mock --chroot -- {command_list}`, the `--` separator might be the problem here. Here's the command: ``` Running in chroot: ['--', '/bin/sh', '-c', '{ rm -f /var/lib/rpm/__db*; rm -rf /var/cache/yum/*; set -x; rm -rf /mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/buildinstall/Server && lorax --product=Fedora --version=Rawhide --release=Rawhide --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-swig-4.0-3620210916145835-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-IO-stringy-2.113-3620210920105958-2 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-postgresql-11-3620210824092915-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-postgresql-10-3620211019194202-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-postgresql-12-3620210816122311-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-libwww-perl-6.48-3620210903113609-81d42e4a --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-XML-Parser-2.46-3620210907103304-1 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-IO-Socket-SSL-2.068-3620210903104856-7b91f26c --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-postgresql-14-3620211013112805-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-YAML-1.30-3620210906131152-1 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-FCGI-0.79-3620210907081151-1 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-Date-Manip-6.83-3620210903123042-1 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-DBI-1.643-3620210903124335-1 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-DBD-Pg-3.14-3620210906120916-1 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-CGI-4.51-3620211004091221-1 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-DBD-SQLite-1.66-3620210906121309-1 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-App-cpanminus-1.7044-3620210907100605-1 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-openmpi-4.0-3620211005013734-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-DBD-MySQL-4.050-3620210906094406-1 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-nodejs-16-3620211108153954-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-nodejs-14-3620211014121918-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-nginx-mainline-3620211102232012-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-nginx-1.20-3620210908185433-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-dwm-latest-3620201101015024-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-mysql-8.0-3620211031142409-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-dwm-6.2-3620210202111826-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-dwm-6.1-3620210201221442-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-dwm-6.0-3620210201213909-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-cri-o-1.22-3620211112151636-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-cri-o-1.21-3620210818145024-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-avocado-vt-latest-3620210902083930-f4f95988 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-avocado-vt-82lts-3620210902113311-a47ea332 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-perl-5.32-3620210922083143-553f8e7a --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-ghc-9.2-3620211101111632-d099bf28 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/module-ghc-8.10-3620210920031153-5e5ad4a0 --source=file:///mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/repo/f36 --source=https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/comps_repo_Server --variant=Server --noupgrade --buildarch=x86_64 --volid=Fedora-S-dvd-x86_64-rawh --logfile=/mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/logs/x86_64/buildinstall-Server-logs/lorax.log --rootfs-size=3 --squashfs-only /mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/buildinstall/Server ; EXIT_CODE=$? ; chmod -R a+r /mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/buildinstall/Server /mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/logs/x86_64/buildinstall-Server-logs ; chown -R 0 /mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/work/x86_64/buildinstall/Server /mnt/koji/compose/rawhide/Fedora-Rawhide-20211118.n.0/logs/x86_64/buildinstall-Server-logs ; exit $EXIT_CODE; } < /dev/null 2>&1 | /usr/bin/tee /builddir/runroot.log; exit ${PIPESTATUS[0]}'] ```
So what's happening here is: 1. pungi calls `koji runroot` - see https://pagure.io/pungi/blob/master/f/pungi/wrappers/kojiwrapper.py#_105 and https://pagure.io/pungi/blob/master/f/pungi/wrappers/kojiwrapper.py#_88 2. `koji runroot` calls `mock --chroot -- (commands)` - see https://pagure.io/koji/blob/master/f/plugins/builder/runroot.py#_223 That `--` is documented as allowable in the mock manpage: --chroot [--] COMMAND [ARGS...] Run the specified command non-interactively within the chroot (no --clean is performed). but I think with the argparse port we are not properly ignoring it. The argparse port did handle this issue at the level of the main parser - so `mock -- something` would be dealt with - but I think not at the level of the subparser.
well, hm, now I grok it a bit harder, the code in mock that tries to deal with this sort of *ought* to deal with it in that case, because `args` should contain all arguments that are unknown to the parser, and `--` should be the first of those (as `--chroot` *is* known to the parser). So there may be more to this. There may be *another* unknown argument in the command koji constructs, somehow. Let me look a bit harder.
Hum. So, yeah, I still can't quite see why this is happening. But a couple of notes: 1. I can reproduce the problem by adding *another* `--`, as we only handle *one*. 2. We only actually error out here with `--old-chroot` (that is, `--isolation=simple`). With nspawn isolation, even if we get a bare `--` into the command list, mock exits successfully. That is: * `mock -r fedora-rawhide-x86_64 --chroot -- -- ls` shows INFO: Running in chroot: ['--', 'ls'], but doesn't error out - we see the output of the `ls` command, the exit status is 0, and there's no apparent error message from the attempt to run `--` * `mock -r fedora-rawhide-x86_64 --old-chroot --chroot -- -- ls` shows INFO: Running in chroot: ['--', 'ls'] , then crashes with `FileNotFoundError: [Errno 2] No such file or directory: '--'` I don't see where an extra `--` would be coming in to trigger this, though. My attempt to reproduce what koji actually does is this: #!/bin/python3 import os cmd = ["/usr/bin/mock", "-r", "fedora-rawhide-x86_64", "--arch", "x86_64", "--old-chroot", "--chroot", "--", "ls"] os.execvp(cmd[0], cmd) but that doesn't fail, it works OK. I cannot see anywhere we'd be getting an extra "--" from, either.
Aha. I think https://pagure.io/koji/pull-request/3130 would actually fix this, and explains why I couldn't see why it was failing (as I was looking at the *current* koji code, i.e. *with that PR merged*). Previous to that PR, koji was doing this: ["/usr/bin/mock", "-r", "fedora-rawhide-x86_64", "--arch", "x86_64", "--old-chroot", "chroot", "--", "ls"] and that effectively sandbags the code that tries to handle the `--`, because `args[0]` is now "chroot", not "--". This should be fixable in mock by swapping "# handle old-style commands" to happen before "# Optparse.parse_args() eats '--' argument, while argparse doesn't. Do it manually."
so deploying *either one or both* of the koji and mock changes on the builders should resolve this.
FEDORA-EPEL-2021-d5b39e1cc0 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-d5b39e1cc0
FEDORA-2021-2ca5dcfa9e has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-2ca5dcfa9e
FEDORA-2021-1c063ee673 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-1c063ee673
FEDORA-2021-4f0e4d759c has been pushed to the Fedora 34 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-4f0e4d759c` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-4f0e4d759c See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-EPEL-2021-d5b39e1cc0 has been pushed to the Fedora EPEL 7 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-d5b39e1cc0 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2021-2ca5dcfa9e has been pushed to the Fedora 33 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-2ca5dcfa9e` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-2ca5dcfa9e See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-EPEL-2021-5f4f9c022a has been pushed to the Fedora EPEL 8 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-5f4f9c022a See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2021-1c063ee673 has been pushed to the Fedora 35 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-1c063ee673` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-1c063ee673 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2021-1c063ee673 has been pushed to the Fedora 35 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-EPEL-2021-5f4f9c022a has been pushed to the Fedora EPEL 8 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2021-4f0e4d759c has been pushed to the Fedora 34 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-EPEL-2021-d5b39e1cc0 has been pushed to the Fedora EPEL 7 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2021-2ca5dcfa9e has been pushed to the Fedora 33 stable repository. If problem still persists, please make note of it in this bug report.