Bug 1933506 - Building locally packages in mock for x86_64 architecture under Rawhide is not working now
Summary: Building locally packages in mock for x86_64 architecture under Rawhide is no...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Fedora
Classification: Fedora
Component: mock
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Copr Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-02-28 19:52 UTC by Mikhail
Modified: 2021-03-31 12:14 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-03-31 12:14:30 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
mock output (15.90 KB, application/x-xz)
2021-03-03 23:01 UTC, Mikhail
no flags Details
mock output (59.86 KB, text/plain)
2021-03-16 15:47 UTC, Mikhail
no flags Details

Description Mikhail 2021-02-28 19:52:53 UTC
Description of problem:
Building locally packages in mock for x86_64 architecture under Rawhide is not working cause of couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64. 

Some examples:
Errors during downloading metadata for repository 'fedora':
  - Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64 []
Error: Failed to download metadata for repo 'fedora': Cannot prepare internal mirrorlist: Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64 []
Finish: build setup for kernel-5.12.0-0.rc0.20210225gitc03c21ba6f4e.160.fc35.src.rpm
Finish: build phase for kernel-5.12.0-0.rc0.20210225gitc03c21ba6f4e.160.fc35.src.rpm
ERROR: Exception(/home/mikhail/packaging-work/kernel/kernel-5.12.0-0.rc0.20210225gitc03c21ba6f4e.160.fc35.src.rpm) Config(fedora-rawhide-x86_64) 0 minutes 14 seconds
INFO: Results and/or logs in: /home/mikhail/packaging-work/kernel/results_kernel/5.12.0/0.rc0.20210225gitc03c21ba6f4e.160.fc35
INFO: Cleaning up build root ('cleanup_on_failure=True')
Start: clean chroot
Finish: clean chroot


Errors during downloading metadata for repository 'fedora':
  - Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64 []
Error: Failed to download metadata for repo 'fedora': Cannot prepare internal mirrorlist: Curl error (7): Couldn't connect to server for https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64 []
Finish: build setup for kernel-5.12.0-0.rc0.20210225gitc03c21ba6f4e.160.fc35.src.rpm
Finish: build phase for kernel-5.12.0-0.rc0.20210225gitc03c21ba6f4e.160.fc35.src.rpm
ERROR: Exception(/home/mikhail/packaging-work/kernel/kernel-5.12.0-0.rc0.20210225gitc03c21ba6f4e.160.fc35.src.rpm) Config(fedora-rawhide-x86_64) 0 minutes 14 seconds
INFO: Results and/or logs in: /home/mikhail/packaging-work/kernel/results_kernel/5.12.0/0.rc0.20210225gitc03c21ba6f4e.160.fc35
INFO: Cleaning up build root ('cleanup_on_failure=True')
Start: clean chroot
Finish: clean chroot

Comment 1 Pavel Raiskup 2021-03-01 13:02:07 UTC
This is not reproducible on my host, and it seems like some networking connection problems
on your host where you run the mock tool.

Simply speaking, mirrors.fedoraproject.org should be accessible mostly everywhere:

mirrors.fedoraproject.org is an alias for wildcard.fedoraproject.org.
wildcard.fedoraproject.org has address 152.19.134.142
wildcard.fedoraproject.org has address 8.43.85.67
wildcard.fedoraproject.org has address 140.211.169.206
wildcard.fedoraproject.org has address 209.132.190.2
wildcard.fedoraproject.org has address 18.159.254.57
wildcard.fedoraproject.org has address 67.219.144.68
wildcard.fedoraproject.org has address 38.145.60.20
wildcard.fedoraproject.org has address 85.236.55.6
wildcard.fedoraproject.org has address 18.185.136.17
wildcard.fedoraproject.org has address 18.133.140.134
wildcard.fedoraproject.org has address 38.145.60.21
wildcard.fedoraproject.org has address 185.141.165.254
wildcard.fedoraproject.org has address 152.19.134.198
wildcard.fedoraproject.org has IPv6 address 2a05:d014:10:7803:f774:4d7c:e277:a457
wildcard.fedoraproject.org has IPv6 address 2001:4178:2:1269::fed2
wildcard.fedoraproject.org has IPv6 address 2610:28:3090:3001:dead:beef:cafe:fed3
wildcard.fedoraproject.org has IPv6 address 2a05:d01c:c6a:cc01:269:da52:9ae1:43e6
wildcard.fedoraproject.org has IPv6 address 2620:52:3:1:dead:beef:cafe:fed6
wildcard.fedoraproject.org has IPv6 address 2605:bc80:3010:600:dead:beef:cafe:fed9
wildcard.fedoraproject.org has IPv6 address 2620:52:3:1:dead:beef:cafe:fed7
wildcard.fedoraproject.org has IPv6 address 2604:1580:fe00:0:dead:beef:cafe:fed1

If it wasn't, most of the existing Fedora machines out in the wild wouldn't be updated.

Comment 2 Mikhail 2021-03-02 21:59:12 UTC
I don't know how to check connectivity inside mock container, but on host system (outside mock container) curl can connect to server for https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64.


[mikhail@localhost mesa-git]$ curl https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64
[1] 1987020
[mikhail@localhost mesa-git]$ <?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" xmlns="http://www.metalinker.org/" type="dynamic" pubdate="Tue, 02 Mar 2021 21:46:43 GMT" generator="mirrormanager" xmlns:mm0="http://fedorahosted.org/mirrormanager">
<!-- # either path=, or repo= and arch= must be specified
-->
</metalink>

[1]+  Done                    curl https://mirrors.fedoraproject.org/metalink?repo=rawhide

Comment 3 Pavel Raiskup 2021-03-03 07:00:02 UTC
Please try:
$ # full cleanup
$ mock -r fedora-rawhide-x86_64 --scrub=all
$ # retry #1
$ mock --verbose -r fedora-rawhide-x86_64 ...
$ # retry #2
$ mock --verbose -r fedora-rawhide-x86_64 --isolation=simple ...
$ # compress and upload this output
$ mock --verbose -r fedora-rawhide-x86_64 --debug-config

Comment 4 Mikhail 2021-03-03 23:01:01 UTC
Created attachment 1760542 [details]
mock output

Comment 5 Pavel Raiskup 2021-03-04 08:03:02 UTC
Correct variant is:
$ curl "https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64"
(note the quotation).

I'm not sure what is going on here.  I am on F34 as well, and I have no problems
with the network in bootstrap chroot.  That said, I bet that
`--no-bootstrap-chroot` will work-around your problems.

I can not reproduce this though, so I can't help

$ rpm -q kernel systemd
kernel-5.11.2-300.fc34.x86_64
systemd-248~rc2-1.fc34.x86_64

Comment 6 Dale Turner 2021-03-07 17:01:10 UTC
I am getting this with rawhide as well. I tried "-r fedora-34-x86_64", "-r fedora-33-x86_64", and "--isolation=simple". Sometimes mock might work once. Usually not at all. Using "--no-bootstrap-chroot" seems to work.

Comment 7 Ondřej Budai 2021-03-10 17:13:38 UTC
I'm getting the same bug on Fedora 34.

Comment 8 Pavel Raiskup 2021-03-10 17:30:31 UTC
The systemd-resolved has bitten us several times, do you have sane values
in /etc/resolv.conf?

Comment 9 Dale Turner 2021-03-13 19:36:26 UTC
I believe my /etc/resolv.conf is default. 

Is it possible this bug could be related to networking issues for guests in qemu/virt-manager?

Thanks.

Comment 10 Pavel Raiskup 2021-03-14 12:44:39 UTC
> Is it possible this bug could be related to networking issues for guests in qemu/virt-manager?

We are not using virtual machines for the builds ... in mock we build against the
host kernel.

>  I believe my /etc/resolv.conf is default.

Can you check contents of:
/var/lib/mock/fedora-rawhide-x86_64-bootstrap/root/etc/resolv.conf

Prehaps you could try stuff like:
sudo /usr/bin/systemd-nspawn -q --ephemeral -D /var/lib/mock/fedora-rawhide-x86_64-bootstrap/root

And try to debug what is the networking problem there.

Comment 11 Mikhail 2021-03-15 17:42:56 UTC
(In reply to Pavel Raiskup from comment #10)
> Prehaps you could try stuff like:
> sudo /usr/bin/systemd-nspawn -q --ephemeral -D
> /var/lib/mock/fedora-rawhide-x86_64-bootstrap/root
> 
> And try to debug what is the networking problem there.

[mikhail@localhost kernel]$ curl https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64
[1] 17093
[mikhail@localhost kernel]$ <?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" xmlns="http://www.metalinker.org/" type="dynamic" pubdate="Mon, 15 Mar 2021 17:42:06 GMT" generator="mirrormanager" xmlns:mm0="http://fedorahosted.org/mirrormanager">
<!-- # either path=, or repo= and arch= must be specified
-->
</metalink>

[1]+  Done                    curl https://mirrors.fedoraproject.org/metalink?repo=rawhide
[mikhail@localhost kernel]$ 
[mikhail@localhost kernel]$ 
[mikhail@localhost kernel]$ sudo /usr/bin/systemd-nspawn -q --ephemeral -D /var/lib/mock/fedora-rawhide-x86_64-bootstrap/root
[root@root-8412c4a7f96ff8c3 ~]# curl https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64
[1] 12
[root@root-8412c4a7f96ff8c3 ~]# curl: (7) Couldn't connect to server

[1]+  Exit 7                  curl https://mirrors.fedoraproject.org/metalink?repo=rawhide
[root@root-8412c4a7f96ff8c3 ~]#

Comment 12 Mikhail 2021-03-15 18:38:19 UTC
[mikhail@localhost ~]$ sudo /usr/bin/systemd-nspawn -q --ephemeral -D /var/lib/mock/fedora-rawhide-x86_64-bootstrap/root
[sudo] password for mikhail: 
[root@root-eea242f983838f32 ~]# curl https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64
[1] 12
[root@root-eea242f983838f32 ~]# <?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" xmlns="http://www.metalinker.org/" type="dynamic" pubdate="Mon, 15 Mar 2021 18:36:46 GMT" generator="mirrormanager" xmlns:mm0="http://fedorahosted.org/mirrormanager">
<!-- # either path=, or repo= and arch= must be specified
-->
</metalink>

[1]+  Done                    curl https://mirrors.fedoraproject.org/metalink?repo=rawhide
[root@root-eea242f983838f32 ~]# curl https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64
[1] 14
[root@root-eea242f983838f32 ~]# curl: (7) Couldn't connect to server

[1]+  Exit 7                  curl https://mirrors.fedoraproject.org/metalink?repo=rawhide
[root@root-eea242f983838f32 ~]# curl https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64
[1] 16
[root@root-eea242f983838f32 ~]# curl: (7) Couldn't connect to server

[1]+  Exit 7                  curl https://mirrors.fedoraproject.org/metalink?repo=rawhide
[root@root-eea242f983838f32 ~]# curl https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=i686  
[1] 18
[root@root-eea242f983838f32 ~]# curl: (7) Couldn't connect to server

[1]+  Exit 7                  curl https://mirrors.fedoraproject.org/metalink?repo=rawhide
[root@root-eea242f983838f32 ~]# curl https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=i686
[1] 20
[root@root-eea242f983838f32 ~]# curl: (7) Couldn't connect to server

[1]+  Exit 7                  curl https://mirrors.fedoraproject.org/metalink?repo=rawhide
[root@root-eea242f983838f32 ~]# curl https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=i686
[1] 22
[root@root-eea242f983838f32 ~]# curl: (7) Couldn't connect to server

[1]+  Exit 7                  curl https://mirrors.fedoraproject.org/metalink?repo=rawhide
[root@root-eea242f983838f32 ~]# curl https://mirrors.fedoraproject.org                                
curl: (7) Couldn't connect to server
[root@root-eea242f983838f32 ~]# 
[root@root-eea242f983838f32 ~]# curl https://fedoraproject.org
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="https://getfedora.org/">here</a>.</p>
<hr>
<address>Apache Server at fedoraproject.org Port 443</address>
</body></html>
[root@root-eea242f983838f32 ~]# curl https://mirrors.fedoraproject.org
curl: (7) Couldn't connect to server

Comment 13 Miroslav Suchý 2021-03-15 19:25:43 UTC
You have to quote the string:
 curl 'https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64'

But when I was trying to reproduce this I run:

sudo /usr/bin/systemd-nspawn -q --ephemeral -D /var/lib/mock/fedora-rawhide-x86_64-bootstrap/root
[root@root-d15480cb77e0a409 ~]# curl 'https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64'
<?xml version="1.0" encoding="utf-8"?>
....
# ^ this worked
# then several variation of curl without quoting..... but then ... withing let say 10 seconds I got:
[root@root-d15480cb77e0a409 ~]# curl 'https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64'
curl: (7) Couldn't connect to server



I can safely reproduce it. When I scrub the chroot and immediately after creating I can run:

# ls -l /etc/resolv.conf ; cat /etc/resolv.conf ; curl 'https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64' >/dev/null
-rw-r--r--. 1 root 135 933 Mar 15 08:48 /etc/resolv.conf
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search lan redhat.com
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5233  100  5233    0     0  11040      0 --:--:-- --:--:-- --:--:-- 11016



but after several minutes:

# ls -l /etc/resolv.conf ; cat /etc/resolv.conf ; curl 'https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64' >/dev/null
-rw-r--r--. 1 root root 0 Mar 15 20:23 /etc/resolv.conf
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: mirrors.fedoraproject.org

Comment 14 Pavel Raiskup 2021-03-15 20:56:30 UTC
Meh, sorry, I forgot to tell you that `--resolv-conf=off` should be used.  At least
this is what mock uses -- it allows us to use the resolv.conf file pre-prepared
by Mock with nspawn.

So two more questions,
1) can you debug?
sudo /usr/bin/systemd-nspawn -q --ephemeral -D /var/lib/mock/fedora-rawhide-x86_64-bootstrap/root --resolv-conf=off

2) can you please check that mock runs the systemd-nspawn with `--resolv-conf=off` argument?
you should see something like:
/usr/bin/systemd-nspawn -q -M f12f30f64d9c44758a8b3ca557f2b57a -D /var/lib/mock/fedora-rawhide-x86_64-bootstrap/root -a --capability=cap_ipc_lock --bind=/tmp/mock-resolv.j0s0f7z3:/etc/resolv.conf --console=pipe --setenv=TERM=vt100 --setenv=SHELL=/bin/bash --setenv=HOME=/var/lib/mock/fedora-rawhide-x86_64/root/installation-homedir --setenv=HOSTNAME=mock --setenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --setenv=PROMPT_COMMAND=printf "\033]0;<mock-chroot>\007" --setenv=PS1=<mock-chroot> \s-\v\$  --setenv=LANG=C.UTF-8 --setenv=LC_MESSAGES=C.UTF-8 --setenv=LD_PRELOAD=/var/tmp/tmp.mock.j3axoet3/$LIB/nosync.so --resolv-conf=off /usr/bin/dnf --installroot /var/lib/mock/fedora-rawhide-x86_64/root/ --releasever 35 --setopt=deltarpm=False --allowerasing --disableplugin=local --disableplugin=spacewalk install @buildsys-build --setopt=tsflags=nocontexts

3) If answer to 2) is negative, please upload output of `systemd-nspawn --help && rpm -q systemd-container`.

Comment 15 Mikhail 2021-03-15 21:13:36 UTC
(In reply to Pavel Raiskup from comment #14)
> Meh, sorry, I forgot to tell you that `--resolv-conf=off` should be used. 
> At least
> this is what mock uses -- it allows us to use the resolv.conf file
> pre-prepared
> by Mock with nspawn.
> 
> So two more questions,
> 1) can you debug?
> sudo /usr/bin/systemd-nspawn -q --ephemeral -D
> /var/lib/mock/fedora-rawhide-x86_64-bootstrap/root --resolv-conf=off

[mikhail@localhost mesa-git]$ sudo /usr/bin/systemd-nspawn -q --ephemeral -D /var/lib/mock/fedora-rawhide-x86_64-bootstrap/root --resolv-conf=off
[root@root-fa5e3d095b3c881b ~]# curl 'https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64'
curl: (7) Couldn't connect to server
[root@root-fa5e3d095b3c881b ~]# curl 'https://fedoraproject.org'
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="https://getfedora.org/">here</a>.</p>
<hr>
<address>Apache Server at fedoraproject.org Port 443</address>
</body></html>
[root@root-fa5e3d095b3c881b ~]# curl 'https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64'
curl: (7) Couldn't connect to server


> 2) can you please check that mock runs the systemd-nspawn with
> `--resolv-conf=off` argument?
> you should see something like:
> /usr/bin/systemd-nspawn -q -M f12f30f64d9c44758a8b3ca557f2b57a -D
> /var/lib/mock/fedora-rawhide-x86_64-bootstrap/root -a
> --capability=cap_ipc_lock --bind=/tmp/mock-resolv.j0s0f7z3:/etc/resolv.conf
> --console=pipe --setenv=TERM=vt100 --setenv=SHELL=/bin/bash
> --setenv=HOME=/var/lib/mock/fedora-rawhide-x86_64/root/installation-homedir
> --setenv=HOSTNAME=mock --setenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin
> --setenv=PROMPT_COMMAND=printf "\033]0;<mock-chroot>\007"
> --setenv=PS1=<mock-chroot> \s-\v\$  --setenv=LANG=C.UTF-8
> --setenv=LC_MESSAGES=C.UTF-8
> --setenv=LD_PRELOAD=/var/tmp/tmp.mock.j3axoet3/$LIB/nosync.so
> --resolv-conf=off /usr/bin/dnf --installroot
> /var/lib/mock/fedora-rawhide-x86_64/root/ --releasever 35
> --setopt=deltarpm=False --allowerasing --disableplugin=local
> --disableplugin=spacewalk install @buildsys-build --setopt=tsflags=nocontexts

Yes, I see `--resolv-conf=off` argument when start mock.

[mikhail@localhost mesa-git]$ ps aux | grep systemd-nspawn
root       60238  0.0  0.0  17588  8768 ?        Ss   02:11   0:00 /usr/bin/systemd-nspawn -q -M a913009b16da4ff5a89a85e677750886 -D /var/lib/mock/fedora-rawhide-x86_64-bootstrap/root -a --capability=cap_ipc_lock --bind=/tmp/mock-resolv.kkz9rnix:/etc/resolv.conf --console=pipe --setenv=TERM=vt100 --setenv=SHELL=/bin/bash --setenv=HOME=/var/lib/mock/fedora-rawhide-x86_64/root/installation-homedir --setenv=HOSTNAME=mock --setenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --setenv=PROMPT_COMMAND=printf "\033]0;<mock-chroot>\007" --setenv=PS1=<mock-chroot> \s-\v\$  --setenv=LANG=C.UTF-8 --setenv=LC_MESSAGES=C.UTF-8 --resolv-conf=off /usr/bin/dnf --installroot /var/lib/mock/fedora-rawhide-x86_64/root/ -y --releasever 35 --setopt=deltarpm=False --allowerasing --disableplugin=local --disableplugin=spacewalk update --setopt=tsflags=nocontexts
mikhail    60261  0.0  0.0 221520  2216 pts/21   R+   02:11   0:00 grep --color=auto systemd-nspawn

Comment 16 Pavel Raiskup 2021-03-15 21:28:38 UTC
I'm looking at the log, and there's the curl error (7) which indeed isn't caused by
dns problems, that would be error (6) I think.

I probably need you to debug why networking isn't working there.  That can be anything.
On my host, from top of my head:

 $ rpm -q systemd-container kernel selinux-policy-targeted container-selinux
 systemd-container-248~rc2-6.fc34.x86_64
 kernel-5.11.6-300.fc34.x86_64
 selinux-policy-targeted-3.14.7-25.fc34.noarch
 container-selinux-2.158.0-1.gite78ac4f.fc34.noarch

Do you have the same set?

Perhaps you could
upload the /var/cache/mock/fedora-rawhide-x86_64-bootstrap/root_cache/cache.tar.gz
file somewhere so we could use your cache to pre-populate the bootstrap.

Thank you (all) for the additional info!

Comment 17 Pavel Raiskup 2021-03-15 21:43:28 UTC
(In reply to Dale Turner from comment #6)
> I am getting this with rawhide as well. I tried "-r fedora-34-x86_64", "-r
> fedora-33-x86_64", and "--isolation=simple". Sometimes mock might work once.
> Usually not at all. Using "--no-bootstrap-chroot" seems to work.

Dale, do I get it right that you can reproduce this even with --isolation=simple?

Comment 18 Mikhail 2021-03-15 21:48:48 UTC
(In reply to Pavel Raiskup from comment #16)
> I'm looking at the log, and there's the curl error (7) which indeed isn't
> caused by
> dns problems, that would be error (6) I think.
> 
> I probably need you to debug why networking isn't working there.  That can
> be anything.
> On my host, from top of my head:
> 
>  $ rpm -q systemd-container kernel selinux-policy-targeted container-selinux
>  systemd-container-248~rc2-6.fc34.x86_64
>  kernel-5.11.6-300.fc34.x86_64
>  selinux-policy-targeted-3.14.7-25.fc34.noarch
>  container-selinux-2.158.0-1.gite78ac4f.fc34.noarch
> 
> Do you have the same set?

Of course Rawhide has different set.

$ rpm -q systemd-container kernel selinux-policy-targeted container-selinux
systemd-container-248~rc3-1.fc35.x86_64
kernel-5.12.0-0.rc2.20210309git144c79ef3353.166.fc35.x86_64
kernel-5.12.0-0.rc2.20210310git05a59d79793d.168.fc35.x86_64
kernel-5.12.0-0.rc3.170.fc35.x86_64
selinux-policy-targeted-3.14.8-6.fc35.noarch
container-selinux-2.158.0-5.dev.gite78ac4f.fc35.noarch

> 
> Perhaps you could
> upload the
> /var/cache/mock/fedora-rawhide-x86_64-bootstrap/root_cache/cache.tar.gz
> file somewhere so we could use your cache to pre-populate the bootstrap.

https://drive.google.com/file/d/1UZsi-YxsObaUDBKkHmPsdLGGq2FfMJ9x/view?usp=sharing

P.S. "--no-bootstrap-chroot" helps me too.

Comment 19 Pavel Raiskup 2021-03-16 11:56:33 UTC
I tried this on a fresh Rawhide box, and I can't reproduce:
$ rpm -q systemd-container kernel selinux-policy-targeted container-selinux
systemd-container-248~rc3-1.fc35.x86_64
kernel-5.12.0-0.rc2.20210309git144c79ef3353.166.fc35.x86_64
selinux-policy-targeted-3.14.8-6.fc35.noarch
container-selinux-2.158.0-5.dev.gite78ac4f.fc35.noarch

I'd tend to switch this against systemd-container component, but I'm still not
sure whether this is related to --isolation=simple.

Comment 20 Mikhail 2021-03-16 12:04:33 UTC
(In reply to Pavel Raiskup from comment #19)
> I tried this on a fresh Rawhide box, and I can't reproduce:

Have you tried running the mock build twice in a row?

Comment 21 Pavel Raiskup 2021-03-16 13:30:09 UTC
I did, several times, but it shouldn't matter ... as I understand the
problems from previous discussion, dnf even fails to perform the initial
chroot installation (ie `mock -r <chroot> --scrub=all` doesn't help).

Comment 22 Mikhail 2021-03-16 15:47:30 UTC
Created attachment 1763693 [details]
mock output

(In reply to Pavel Raiskup from comment #21)
> I did, several times, but it shouldn't matter ... as I understand the
> problems from previous discussion, dnf even fails to perform the initial
> chroot installation (ie `mock -r <chroot> --scrub=all` doesn't help).

Yes `$ mock -r fedora-rawhide-x86_64 --scrub=all` doesn't help

Comment 23 Pavel Raiskup 2021-03-16 17:40:19 UTC
Ok, I reproduced the problem on a slightly outdated Rawhide, but the problems
disappeared after 'dnf update' on the host:

...
Dependencies resolved.
========================================================================...
 Package                            Architecture        Version         ...
========================================================================...
Upgrading:
 acl                                x86_64              2.3.0-1.fc35    ...
 binutils                           x86_64              2.36.1-7.fc35   ...
 binutils-gold                      x86_64              2.36.1-7.fc35   ...
 gnutls                             x86_64              3.7.1-1.fc35    ...
 libacl                             x86_64              2.3.0-1.fc35    ...
 libattr                            x86_64              2.5.0-1.fc35    ...
 libnfsidmap                        x86_64              1:2.5.3-2.fc35  ...
 nfs-utils                          x86_64              1:2.5.3-2.fc35  ...
 redhat-rpm-config                  noarch              181-1.fc35      ...
 rng-tools                          x86_64              6.12-1.fc35     ...
 selinux-policy                     noarch              3.14.8-6.fc35   ...
 selinux-policy-mls                 noarch              3.14.8-6.fc35   ...
 selinux-policy-targeted            noarch              3.14.8-6.fc35   ...
 xfsprogs                           x86_64              5.11.0-1.fc35   ...
....

I don't know if that is a coincidence or not.  I'll try once more once I have
a bit more time...

Comment 24 Mikhail 2021-03-17 22:22:03 UTC
With glib2 2.67.6-1.fc35 I started reproduce this issue also on host system (outside mock container)
https://bugzilla.redhat.com/show_bug.cgi?id=1940246

Comment 25 Dale Turner 2021-03-21 15:06:31 UTC
I'm not sure what has changed, but I seem to be able to use mock now without "--no-bootstrap-chroot".

Just to be clear, previously, mock did NOT work with "--isolation=simple". Sorry for the delay in responding.

Thanks to everyone.

Comment 26 Pavel Raiskup 2021-03-31 12:14:30 UTC
Looks like fixed then?  Closing, please reopen if you still see the issue
with updated F34 or F35.


Note You need to log in before you can comment on or make changes to this bug.