Bug 1450608

Summary: FTBFS: python-requests cannot be built with mock-1.4.1+
Product: [Fedora] Fedora Reporter: Stephen Gallagher <sgallagh>
Component: python-requestsAssignee: Fedora Infrastructure SIG <infra-sig>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: rawhideCC: cstratak, infra-sig, jcline, jdisnard, jeremy, mebrown, msimacek, msuchy, praiskup, rbean, sagarun, williams
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-07-26 10:35:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1400162    
Attachments:
Description Flags
%check error output none

Description Stephen Gallagher 2017-05-13 21:51:30 UTC
Created attachment 1278479 [details]
%check error output

Description of problem:
python-requests encounters test failures in %check when building against a current Fedora 26 buildroot. See attached file for errors

Version-Release number of selected component (if applicable):
python-requests-2.13.0-1.fc26

How reproducible:
Every time

Steps to Reproduce:
1. fedpkg clone python-requests
2. fedpkg switch-branch f26 (probably also happens on Rawhide)
3. fedpkg mock-build

Actual results:
Tests fail in %check, so the build does not complete

Expected results:
Package builds successfully

Additional info:
This is blocking the Fedora Modularity effort.

Comment 1 Stephen Gallagher 2017-05-13 21:52:37 UTC
Whoops, that attachment was from my Rawhide mock-build. That said, it's showing the same results as on F26. So it clearly affects F26+.

Comment 2 Stephen Gallagher 2017-05-14 09:36:18 UTC
OK, so this is odd. It fails consistently when I do a local mock-build (on two separate machines), but it succeeds consistently when I do a Koji scratch-build.

Something about these tests apparently likes the Koji builder environment but not the local environment.

Comment 3 Charalampos Stratakis 2017-05-15 12:25:08 UTC
This must be related to the latest mock which uses systemd-nspawn instead of chroot [0]

Using mock-1.3.4-1.fc25.noarch, python-requests builds fine locally, however when I update to mock-1.4.1-1.fc25.noarch from updates-testing I get these test failures.

[0] https://github.com/rpm-software-management/mock/wiki/Release-Notes-1.4.1

Comment 4 Stephen Gallagher 2017-05-15 12:28:43 UTC
Ahh, that would make sense. So should the bug be reassigned to mock or will the fix be in python-requests?

Comment 5 Charalampos Stratakis 2017-05-15 12:49:00 UTC
(In reply to Stephen Gallagher from comment #4)
> Ahh, that would make sense. So should the bug be reassigned to mock or will
> the fix be in python-requests?

Not sure of the internal mechanics of this mock change here and how that might affect test coverage in requests (or any other package). I suppose the bugzilla can re reassigned to mock for the time being as the maintainers should have more insight on this.

Comment 6 Jeremy Cline 2017-05-15 17:23:57 UTC
Ahh, I know what this is. Sorry for the slow response, I wasn't in front of my computer much this weekend.

Can you try building without being connected to the Red Hat VPN? The upstream tests use a "tar pit" IP[1] that the tests assume will always fail with a connection timeout. Red Hat has a host there that refuses the connection and that causes the tests to fail.

[1] https://github.com/kennethreitz/requests/blob/bf1a2a4e399ec5e75c5e9aadae564f77ca99afff/tests/test_requests.py#L40

Comment 7 Jeremy Cline 2017-05-17 16:47:17 UTC
Okay, I confirmed that even without the Red Hat VPN it's still a problem with the new mock.

Specifically, the issue is that previously opening a socket to 10.255.255.1 port 80 would simply hang forever, but now it's getting a "[Errno 101] Network is unreachable". 

The systemd-nspawn command is being run with --private-network[0], which is causing this problem. Running without that works fine. I guess that anything not heading for the loopback device gets reset rather than quietly dropped?

[0] https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html#--private-network

Comment 8 Stephen Gallagher 2017-05-17 17:36:00 UTC
(In reply to Jeremy Cline from comment #7)
> Okay, I confirmed that even without the Red Hat VPN it's still a problem
> with the new mock.
> 
> Specifically, the issue is that previously opening a socket to 10.255.255.1
> port 80 would simply hang forever, but now it's getting a "[Errno 101]
> Network is unreachable". 
> 
> The systemd-nspawn command is being run with --private-network[0], which is
> causing this problem. Running without that works fine. I guess that anything
> not heading for the loopback device gets reset rather than quietly dropped?
> 
> [0]
> https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html#--
> private-network

I would call that a features rather than a bug. Our packaging policy has always required that packages build successfully in an environment with no network access. This is just making it fail hard if we're trying to reach out of the chroot.

Comment 9 Jeremy Cline 2017-05-17 18:07:09 UTC
Sure, I didn't mean to imply it was a bug, just that that's why it stopped working.

I suppose I'll either need to patch these tests out (which isn't very appealing) or re-work them so they only use the loopback device. Given that, I'll go ahead and re-assign this bug to requests.

Comment 10 Igor Gnatenko 2017-07-26 10:35:55 UTC
it seems this bug was fixed by @jcline already.