Bug 1718294 - nbdkit fails to build with Python 3.8
Summary: nbdkit fails to build with Python 3.8
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: nbdkit
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Richard W.M. Jones
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: PYTHON38
TreeView+ depends on / blocked
 
Reported: 2019-06-07 12:34 UTC by Petr Viktorin (pviktori)
Modified: 2019-06-10 10:42 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-10 08:44:01 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Petr Viktorin (pviktori) 2019-06-07 12:34:32 UTC
nbdkit fails to build with Python 3.8.0b1.

Looking at the build, it seems "nbdkit python --dump-plugin" attempts to load a Python extension module directly as a C shared library. This is not supported; Python modules should be imported from Python code or using C API like PyImport_ImportModule.

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.8/fedora-rawhide-x86_64/00927318-nbdkit/

For all our attempts to build nbdkit with Python 3.8, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.8/package/nbdkit/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.8:
https://copr.fedorainfracloud.org/coprs/g/python/python3.8/

Let us know here if you have any questions.

Comment 1 Richard W.M. Jones 2019-06-07 12:46:09 UTC
Your interpretation is wrong, the *.so file isn't a Python module, and we use
PyRun_SimpleFileEx to run the Python code.

The reason it fails to build is some problem in Python 3.8:

> undefined symbol: PyByteArray_Type

I can see a load of people have this problem on-line, but no clear explanation
of why it happens or resolution for it.

Comment 2 Richard W.M. Jones 2019-06-07 16:40:21 UTC
Oh I see, it's because in this case we do need to use python-embed.pc instead
of python.pc.

Comment 3 Richard W.M. Jones 2019-06-09 21:11:28 UTC
Upstream fix:
https://github.com/libguestfs/nbdkit/commit/fb1a74b3d84571bc4c6bb57b26128c92b3368c52

nbdkit >= 1.13.5

Soon to be added to Rawhide.

Comment 4 Miro Hrončok 2019-06-10 07:38:33 UTC
It still fails:

+ echo ./test-dump-plugin.sh: run cleanup hooks: exit code 1
./test-dump-plugin.sh: run cleanup hooks: exit code 1
+ (( i = 0 ))
+ (( i < 0 ))
+ exit 1
FAIL test-dump-plugin.sh (exit status: 1)
SKIP: test-memory-largest.sh
============================
/builddir/build/BUILD/nbdkit-1.13.5/server/nbdkit -v -P memory-largest.pid -U /tmp/tmp.6wzqEvg2nx -- /builddir/build/BUILD/nbdkit-1.13.5/plugins/memory/.libs/nbdkit-memory-plugin.so 9223372036854775807
nbdkit: debug: TLS disabled: could not load TLS certificates
nbdkit: debug: registering plugin /builddir/build/BUILD/nbdkit-1.13.5/plugins/memory/.libs/nbdkit-memory-plugin.so
nbdkit: debug: registered plugin /builddir/build/BUILD/nbdkit-1.13.5/plugins/memory/.libs/nbdkit-memory-plugin.so (name memory)
nbdkit: debug: memory: load
nbdkit: debug: memory: config key=size, value=9223372036854775807
nbdkit: debug: memory: config_complete
nbdkit: debug: bound to unix socket /tmp/tmp.6wzqEvg2nx
nbdkit: debug: forked into background (new pid = 17321)
nbdkit: debug: written pidfile /builddir/build/BUILD/nbdkit-1.13.5/tests/memory-largest.pid
./test-memory-largest.sh: run cleanup hooks: exit code 77
nbdkit: debug: memory: unload
SKIP test-memory-largest.sh (exit status: 77)
SKIP: test-pattern-largest.sh
=============================
/builddir/build/BUILD/nbdkit-1.13.5/server/nbdkit -v -P pattern-largest.pid -U /tmp/tmp.waDs8y95wc -- /builddir/build/BUILD/nbdkit-1.13.5/plugins/pattern/.libs/nbdkit-pattern-plugin.so 9223372036854775807
nbdkit: debug: TLS disabled: could not load TLS certificates
nbdkit: debug: registering plugin /builddir/build/BUILD/nbdkit-1.13.5/plugins/pattern/.libs/nbdkit-pattern-plugin.so
nbdkit: debug: registered plugin /builddir/build/BUILD/nbdkit-1.13.5/plugins/pattern/.libs/nbdkit-pattern-plugin.so (name pattern)
nbdkit: debug: pattern: load
nbdkit: debug: pattern: config key=size, value=9223372036854775807
nbdkit: debug: pattern: config_complete
nbdkit: debug: bound to unix socket /tmp/tmp.waDs8y95wc
nbdkit: debug: forked into background (new pid = 20699)
nbdkit: debug: written pidfile /builddir/build/BUILD/nbdkit-1.13.5/tests/pattern-largest.pid
./test-pattern-largest.sh: run cleanup hooks: exit code 77
nbdkit: debug: pattern: unload
SKIP test-pattern-largest.sh (exit status: 77)
SKIP: test-vddk-real.sh
=======================
+ requires test x '!=' x
+ echo './test-vddk-real.sh: â€test x != x’ failed with error code 1'
./test-vddk-real.sh: â€test x != x’ failed with error code 1
+ echo './test-vddk-real.sh: test prerequisite is missing or not working'
./test-vddk-real.sh: test prerequisite is missing or not working
+ exit 77
+ _run_cleanup_hooks
+ status=77
+ set +e
+ trap '' INT QUIT TERM EXIT ERR
+ echo ./test-vddk-real.sh: run cleanup hooks: exit code 77
./test-vddk-real.sh: run cleanup hooks: exit code 77
+ (( i = 0 ))
+ (( i < 0 ))
+ exit 77
SKIP test-vddk-real.sh (exit status: 77)
FAIL: test-python-exception.sh
==============================
+ test '!' -d .
+ '[' '' = 1 ']'
+ output=test-python-exception.out
+ rm -f test-python-exception.out
+ cleanup_fn rm -f test-python-exception.out
+ _cleanup_hook[${#_cleanup_hook[@]}]='rm -f test-python-exception.out'
+ nbdkit -f -v python ./python-exception.py test=simple
+ :
+ cat test-python-exception.out
/builddir/build/BUILD/nbdkit-1.13.5/server/nbdkit -f -v -- /builddir/build/BUILD/nbdkit-1.13.5/plugins/python/.libs/nbdkit-python-plugin.so ./python-exception.py test=simple
nbdkit: debug: TLS disabled: could not load TLS certificates
nbdkit: error: cannot open plugin '/builddir/build/BUILD/nbdkit-1.13.5/plugins/python/.libs/nbdkit-python-plugin.so': /builddir/build/BUILD/nbdkit-1.13.5/plugins/python/.libs/nbdkit-python-plugin.so: undefined symbol: PyByteArray_Type
Use 'nbdkit --help' or read the nbdkit(1) manual page for documentation.
+ grep 'this is the test string' test-python-exception.out
++ _run_cleanup_hooks
++ status=1
++ set +e
++ trap '' INT QUIT TERM EXIT ERR
++ echo ./test-python-exception.sh: run cleanup hooks: exit code 1
./test-python-exception.sh: run cleanup hooks: exit code 1
++ (( i = 0 ))
++ (( i < 1 ))
++ rm -f test-python-exception.out
++ (( ++i  ))
++ (( i < 1 ))
++ exit 1
FAIL test-python-exception.sh (exit status: 1)
FAIL: test-shebang-python.sh
============================
/builddir/build/BUILD/nbdkit-1.13.5/server/nbdkit -P shebang-python.pid -U /tmp/tmp.CWpqXvwwgD -f -v -- /builddir/build/BUILD/nbdkit-1.13.5/plugins/python/.libs/nbdkit-python-plugin.so ./shebang.py
nbdkit: debug: TLS disabled: could not load TLS certificates
nbdkit: error: cannot open plugin '/builddir/build/BUILD/nbdkit-1.13.5/plugins/python/.libs/nbdkit-python-plugin.so': /builddir/build/BUILD/nbdkit-1.13.5/plugins/python/.libs/nbdkit-python-plugin.so: undefined symbol: PyByteArray_Type
Use 'nbdkit --help' or read the nbdkit(1) manual page for documentation.
./test-shebang-python.sh: PID file was not created
./test-shebang-python.sh: run cleanup hooks: exit code 1
FAIL test-shebang-python.sh (exit status: 1)
SKIP: test-fua.sh
=================
++ mktemp -d
+ sockdir=/tmp/tmp.2UBabllkkp
+ files='fua.img
       fua1.log fua1.pid
       fua2.log fua2.pid
       fua3.log fua3.pid
       fua4.log fua4.pid'
+ rm -f fua.img fua1.log fua1.pid fua2.log fua2.pid fua3.log fua3.pid fua4.log fua4.pid
+ truncate -s 1M fua.img
+ qemu-io -f raw -t none -c flush -c 'w -f -z 0 64k' fua.img
file system may not support O_DIRECT
can't open device fua.img: Could not open 'fua.img': Invalid argument
+ echo './test-fua.sh: missing or broken qemu-io'
./test-fua.sh: missing or broken qemu-io
+ rm fua.img
+ exit 77
+ _run_cleanup_hooks
+ status=77
+ set +e
+ trap '' INT QUIT TERM EXIT ERR
+ echo ./test-fua.sh: run cleanup hooks: exit code 77
./test-fua.sh: run cleanup hooks: exit code 77
+ (( i = 0 ))
+ (( i < 0 ))
+ exit 77
SKIP test-fua.sh (exit status: 77)
RPM build errors:
+ exit 1
BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.U2FYct (%check)
BUILDSTDERR:     Bad exit status from /var/tmp/rpm-tmp.U2FYct (%check)
Child return code was: 1

Comment 5 Richard W.M. Jones 2019-06-10 08:12:46 UTC
Please attach or link to the complete build log.

Comment 7 Richard W.M. Jones 2019-06-10 08:33:48 UTC
checking for PYTHON... no
checking for PYTHON... no
checking for PYTHON... yes

Comparing this to the source:

https://github.com/libguestfs/nbdkit/blob/46323648adb79f338d25f6ee8806249f017d3e38/configure.ac#L476-L508

indicates that the python-embed.pc files were not found.  I suppose it's called python-<VERSION>-embed
not python-embed-<VERSION>.

Comment 8 Richard W.M. Jones 2019-06-10 08:44:01 UTC
My attempts to build this locally in mock always fail with:

[MIRROR] glusterfs-6.2-1.fc31.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 84 but expected size is: 668104
[FAILED] glusterfs-6.2-1.fc31.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success

which seems to be a server side problem.  Anyway I have pushed another patch
blindly which may fix this, in nbdkit-1.13.5-2.fc31.

Comment 9 Miro Hrončok 2019-06-10 10:42:44 UTC
For the server problem, I recommend scrubbing mock dnf cache,  with mock -r ... --scrub=dnf-cache.

Anyway, running another build.


Note You need to log in before you can comment on or make changes to this bug.