Bug 1724753

Summary:	Python 3.7 FTBFS with %source_date_epoch_from_changelog 1 (can't find '__main__' module in .../test_zip.zip)
Product:	[Fedora] Fedora	Reporter:	Miro Hrončok <mhroncok>
Component:	python3	Assignee:	Miro Hrončok <mhroncok>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	rawhide	CC:	cstratak, dmalcolm, m.cyprian, mhroncok, pviktori, rkuska, shcherbina.iryna, slavek.kabrda, tomspur, torsava, vstinner
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
URL:	http://apps.fedoraproject.org/koschei/package/python3
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-07-01 14:37:13 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Miro Hrončok 2019-06-27 17:35:22 UTC

Description of problem:
Package python3 fails to build from source in Fedora rawhide.

Version-Release number of selected component (if applicable):
3.7.4~rc1-1.fc31

Steps to Reproduce:
koji build --scratch f31 python3-3.7.4~rc1-1.fc31.src.rpm

Additional info:
This package is tracked by Koschei. See:
http://apps.fedoraproject.org/koschei/package/python3

https://apps.fedoraproject.org/koschei/build/6603903

expat-devel
2.2.6-2.fc30
2.2.7-1.fc31

 redhat-rpm-config
133-1.fc31
133-2.fc31

 rpm
4.14.90-0.git14653.18...
4.15.0-0.beta.1.fc31

 expat
2.2.6-2.fc30
2.2.7-1.fc31

 rpm-libs
4.14.90-0.git14653.18...
4.15.0-0.beta.1.fc31

 krb5-libs
1.17-31.fc31
1.17-33.fc31

 gawk
5.0.1-1.fc31
5.0.1-2.fc31

 rpm-build
4.14.90-0.git14653.18...
4.15.0-0.beta.1.fc31

 hwdata
0.324-1.fc31
0.325-1.fc31

Failures are all ralted to:

can't find '__main__' module in .../test_zip.zip

Comment 1 Miro Hrončok 2019-06-27 18:12:06 UTC

This is https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/57

Setting SOURCE_DATE_EPOCH breaks the tests:

$ python3.7 -m test  test_cmd_line_script test_multiprocessing_main_handling test_runpy
Run tests sequentially
0:00:00 load avg: 2.43 [1/3] test_cmd_line_script
0:00:02 load avg: 2.43 [2/3] test_multiprocessing_main_handling
0:00:11 load avg: 2.76 [3/3] test_runpy

== Tests result: SUCCESS ==

All 3 tests OK.

Total duration: 12 sec 253 ms
Tests result: SUCCESS



$ SOURCE_DATE_EPOCH=0 python3.7 -m test  test_cmd_line_script test_multiprocessing_main_handling test_runpy
Run tests sequentially
0:00:00 load avg: 2.37 [1/3] test_cmd_line_script
test test_cmd_line_script failed -- Traceback (most recent call last):
  File "/usr/lib64/python3.7/test/test_cmd_line_script.py", line 268, in test_zipfile_compiled
    zipimport.zipimporter)
  File "/usr/lib64/python3.7/test/test_cmd_line_script.py", line 129, in _check_script
    *run_args, __isolated=False, __cwd=cwd, **env_vars
  File "/usr/lib64/python3.7/test/support/script_helper.py", line 157, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/usr/lib64/python3.7/test/support/script_helper.py", line 143, in _assert_python
    res.fail(cmd_line)
  File "/usr/lib64/python3.7/test/support/script_helper.py", line 84, in fail
    err))
AssertionError: Process return code is 1
command line: ['/usr/bin/python3.7', '-X', 'faulthandler', '-E', '/tmp/tmpkyn685py/test_zip.zip', 'test1', 'test2', 'test3']

stdout:
---

---

stderr:
---
/usr/bin/python3.7: can't find '__main__' module in '/tmp/tmpkyn685py/test_zip.zip'
---

0:00:02 load avg: 2.37 [2/3/1] test_multiprocessing_main_handling -- test_cmd_line_script failed
test test_multiprocessing_main_handling failed -- multiple errors occurred; run in verbose mode for details
0:00:09 load avg: 2.16 [3/3/2] test_runpy -- test_multiprocessing_main_handling failed
test test_runpy failed -- Traceback (most recent call last):
  File "/usr/lib64/python3.7/runpy.py", line 153, in _get_module_details
    code = loader.get_code(mod_name)
zipimport.ZipImportError: can't find module '__main__'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib64/python3.7/runpy.py", line 219, in _get_main_module_details
    return _get_module_details(main_name)
  File "/usr/lib64/python3.7/runpy.py", line 155, in _get_module_details
    raise error(format(e)) from e
ImportError: can't find module '__main__'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib64/python3.7/test/test_runpy.py", line 715, in test_zipfile_compiled
    mod_name=mod_name, check_loader=False)
  File "/usr/lib64/python3.7/test/test_runpy.py", line 631, in _check_script
    self.check_code_execution(create_ns, expected_ns)
  File "/usr/lib64/python3.7/test/test_runpy.py", line 130, in check_code_execution
    result_ns = create_namespace(None)
  File "/usr/lib64/python3.7/test/test_runpy.py", line 606, in create_ns
    return run_path(script_name, init_globals)
  File "/usr/lib64/python3.7/runpy.py", line 275, in run_path
    mod_name, mod_spec, code = _get_main_module_details()
  File "/usr/lib64/python3.7/runpy.py", line 223, in _get_main_module_details
    (main_name, sys.path[0])) from exc
ImportError: can't find '__main__' module in '/tmp/tmpp4g2mgp1/test_zip.zip'

test_runpy failed

== Tests result: FAILURE ==

3 tests failed:
    test_cmd_line_script test_multiprocessing_main_handling test_runpy

Total duration: 9 sec 896 ms
Tests result: FAILURE



$ SOURCE_DATE_EPOCH=$(date +%s) python3.7 -m test  test_cmd_line_script test_multiprocessing_main_handling test_runpy
Run tests sequentially
0:00:00 load avg: 1.91 [1/3] test_cmd_line_script
test test_cmd_line_script failed -- Traceback (most recent call last):
  File "/usr/lib64/python3.7/test/test_cmd_line_script.py", line 268, in test_zipfile_compiled
    zipimport.zipimporter)
  File "/usr/lib64/python3.7/test/test_cmd_line_script.py", line 129, in _check_script
    *run_args, __isolated=False, __cwd=cwd, **env_vars
  File "/usr/lib64/python3.7/test/support/script_helper.py", line 157, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/usr/lib64/python3.7/test/support/script_helper.py", line 143, in _assert_python
    res.fail(cmd_line)
  File "/usr/lib64/python3.7/test/support/script_helper.py", line 84, in fail
    err))
AssertionError: Process return code is 1
command line: ['/usr/bin/python3.7', '-X', 'faulthandler', '-E', '/tmp/tmp1xbj03s2/test_zip.zip', 'test1', 'test2', 'test3']

stdout:
---

---

stderr:
---
/usr/bin/python3.7: can't find '__main__' module in '/tmp/tmp1xbj03s2/test_zip.zip'
---

0:00:27 load avg: 1.86 [2/3/1] test_multiprocessing_main_handling -- test_cmd_line_script failed
test test_multiprocessing_main_handling failed -- multiple errors occurred; run in verbose mode for details
0:00:35 load avg: 1.88 [3/3/2] test_runpy -- test_multiprocessing_main_handling failed
test test_runpy failed -- Traceback (most recent call last):
  File "/usr/lib64/python3.7/runpy.py", line 153, in _get_module_details
    code = loader.get_code(mod_name)
zipimport.ZipImportError: can't find module '__main__'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib64/python3.7/runpy.py", line 219, in _get_main_module_details
    return _get_module_details(main_name)
  File "/usr/lib64/python3.7/runpy.py", line 155, in _get_module_details
    raise error(format(e)) from e
ImportError: can't find module '__main__'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib64/python3.7/test/test_runpy.py", line 715, in test_zipfile_compiled
    mod_name=mod_name, check_loader=False)
  File "/usr/lib64/python3.7/test/test_runpy.py", line 631, in _check_script
    self.check_code_execution(create_ns, expected_ns)
  File "/usr/lib64/python3.7/test/test_runpy.py", line 130, in check_code_execution
    result_ns = create_namespace(None)
  File "/usr/lib64/python3.7/test/test_runpy.py", line 606, in create_ns
    return run_path(script_name, init_globals)
  File "/usr/lib64/python3.7/runpy.py", line 275, in run_path
    mod_name, mod_spec, code = _get_main_module_details()
  File "/usr/lib64/python3.7/runpy.py", line 223, in _get_main_module_details
    (main_name, sys.path[0])) from exc
ImportError: can't find '__main__' module in '/tmp/tmpfqqi1p43/test_zip.zip'

test_runpy failed

== Tests result: FAILURE ==

3 tests failed:
    test_cmd_line_script test_multiprocessing_main_handling test_runpy

Total duration: 36 sec 337 ms
Tests result: FAILURE

Comment 2 Miro Hrončok 2019-06-27 18:25:01 UTC

https://src.fedoraproject.org/rpms/python3/pull-request/116

Also, I could not reporduce this in any othe Python version larger or smaller. I've bumped the priorities of Pythons in Koschei to make sure.

Comment 3 Victor Stinner 2019-06-28 09:27:45 UTC

I confirm that test_cmd_line_script test_multiprocessing_main_handling test_runpy fail in Python 3.7 when SOURCE_DATE_EPOCH env var is set. This bug has been fixed in Python 3.8, but not in Python 3.7:
https://bugs.python.org/issue34022#msg346814

Comment 4 Victor Stinner 2019-07-01 14:37:13 UTC

The package build has been fixed by not setting SOURCE_DATE_EPOCH env var:

commit 9f5808cf53efafc3f76c1ed425d5b0bb7818e871
Author: Miro Hrončok <miro>
Date:   Thu Jun 27 20:04:01 2019 +0200

    Don't let RPM set SOURCE_DATE_EPOCH based on the latest %changelog date
    
    It breaks tests with: can't find '__main__' module in .../test_zip.zip
    
    Reported at https://bugs.python.org/issue34022
    Tracked at https://bugzilla.redhat.com/show_bug.cgi?id=1724753

The 3 tests still fails when run explicitly with SOURCE_DATE_EPOCH env var set, but it's expected: that's a new feature which is a work-in-progress. It's not yet fully fixed, even in Python 3.9 (dev version).

See also: https://bugzilla.redhat.com/show_bug.cgi?id=1686078#c5