Bug 1665490 - Importing xmlsec causes segfault when exception message should be displayed [wait for Python 3.7.5]
Summary: Importing xmlsec causes segfault when exception message should be displayed [...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: python3
Version: 29
Hardware: Unspecified
OS: Unspecified
low
unspecified
Target Milestone: ---
Assignee: Victor Stinner
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On: 1665459
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-01-11 15:17 UTC by Jan Pazdziora
Modified: 2019-11-03 00:11 UTC (History)
12 users (show)

Fixed In Version: python3-3.7.5-1.fc30 python3-3.7.5-1.fc29 python3-3.7.5-1.fc31
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1665459
Environment:
Last Closed: 2019-11-02 01:12:40 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Python 37467 0 None None None 2019-07-01 10:37:03 UTC

Description Jan Pazdziora 2019-01-11 15:17:47 UTC
+++ This bug was initially created as a clone of Bug #1665459 +++

Description of problem:

The documentation of python xmlsec module https://github.com/mehcode/python-xmlsec/blob/master/doc/source/examples.rst#sign shows that it should work just fine with lxml. However, when xmlsec module is imported in python code (not even used), attempt to parse second xml file with lxml's etree.parse results in segfault.

Version-Release number of selected component (if applicable):

python3-xmlsec-1.3.3-3.fc29.x86_64
python3-lxml-4.2.5-1.fc29.x86_64

How reproducible:

Deterministic.

Steps to Reproduce:
1. cat > sf.xml
<data/>
Ctrl-D here

2. cat > sf.py
from lxml import etree
import xmlsec
xml1 = etree.parse("sf.xml")
print(xml1)
xml2 = etree.parse("sf.xml")
print(xml2)
Ctrl-D here

3. python3 sf.py

Actual results:

<lxml.etree._ElementTree object at 0x7f39227536c8>
Traceback (most recent call last):
  File "sf.py", line 5, in <module>
    xml2 = etree.parse("sf.xml")
  File "src/lxml/etree.pyx", line 3426, in lxml.etree.parse
  File "src/lxml/parser.pxi", line 1840, in lxml.etree._parseDocument
  File "src/lxml/parser.pxi", line 1866, in lxml.etree._parseDocumentFromURL
  File "src/lxml/parser.pxi", line 1770, in lxml.etree._parseDocFromFile
  File "src/lxml/parser.pxi", line 1163, in lxml.etree._BaseParser._parseDocFromFile
  File "src/lxml/parser.pxi", line 601, in lxml.etree._ParserContext._handleParseResultDoc
  File "src/lxml/parser.pxi", line 711, in lxml.etree._handleParseResult
  File "src/lxml/parser.pxi", line 651, in lxml.etree._raiseParseError
Segmentation fault (core dumped)

Expected results:

<lxml.etree._ElementTree object at 0x7f246c117c08>
<lxml.etree._ElementTree object at 0x7f246c117c88>

Additional info:

When the "import xmlsec" line is commented out, I get the expected result.

This used to work on Fedora 27 with

python3-lxml-4.1.1-1.fc27.x86_64
python3-xmlsec-1.3.3-1.fc27.x86_64

It started to fail on Fedora 28 with

python3-lxml-4.2.5-1.fc28.x86_64
python3-xmlsec-1.3.3-1.fc28.x86_64

and fails up to rawhide.

--- Additional comment from Jan Pazdziora on 2019-01-11 14:30:30 CET ---

Backtrace is:

Core was generated by `python3 sf.py'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  find_maxchar_surrogates (num_surrogates=<synthetic pointer>, 
    maxchar=<synthetic pointer>, 
    end=0xfffffffffffffffd <error: Cannot access memory at address 0xfffffffffffffffd>, begin=0x1 <error: Cannot access memory at address 0x1>)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Objects/unicodeobject.c:1660
1660	            ch = *iter;
Missing separate debuginfos, use: dnf debuginfo-install libgpg-error-1.31-2.fc29.x86_64 libtool-ltdl-2.4.6-25.fc29.x86_64 libxml2-2.9.8-4.fc29.x86_64 libxslt-1.1.32-3.fc29.x86_64 openssl-libs-1.1.1-0.pre9.3.fc29.x86_64 python3-xmlsec-1.3.3-3.fc29.x86_64 xmlsec1-openssl-1.2.25-5.fc29.x86_64 xz-libs-5.2.4-3.fc29.x86_64 zlib-1.2.11-10.fc29.x86_64
(gdb) bt
#0  find_maxchar_surrogates (num_surrogates=<synthetic pointer>, 
    maxchar=<synthetic pointer>, 
    end=0xfffffffffffffffd <error: Cannot access memory at address 0xfffffffffffffffd>, begin=0x1 <error: Cannot access memory at address 0x1>)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Objects/unicodeobject.c:1660
#1  _PyUnicode_Ready (unicode=b'sg.xml')
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Objects/unicodeobject.c:1699
#2  0x00007f7f28965651 in unicode_fromformat_write_str (precision=-1, 
    width=-1, str=<optimized out>, writer=0x7ffd3b6ecab0)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Objects/unicodeobject.c:2909
#3  unicode_fromformat_arg (vargs=0x7ffd3b6ecb10, 
    f=0x7f7f28b34964 "U\", line %d\n", writer=0x7ffd3b6ecab0)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Objects/unicodeobject.c:2801
#4  PyUnicode_FromFormatV (format=<optimized out>, vargs=<optimized out>)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Objects/unicodeobject.c:2908
#5  0x00007f7f289ffd09 in PyUnicode_FromFormat (
    format=format@entry=0x7f7f28b3495b "  File \"%U\", line %d\n")
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Objects/unicodeobject.c:2960
#6  0x00007f7f28b176f1 in print_exception (value=None, 
    f=<_io.TextIOWrapper at remote 0x7f7f28248708>)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Python/pythonrun.c:753
#7  print_exception_recursive (f=<_io.TextIOWrapper at remote 0x7f7f28248708>, 
    value=<XMLSyntaxError(error_log=<lxml.etree._ListErrorLog at remote 0x7f7f279380e8>, code=1) at remote 0x7f7f27999d38>, seen=<optimized out>)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Python/pythonrun.c:901
#8  0x00007f7f28b19eeb in PyErr_Display (exception=<optimized out>, 
    value=<XMLSyntaxError(error_log=<lxml.etree._ListErrorLog at remote 0x7f7f279380e8>, code=1) at remote 0x7f7f27999d38>, tb=<optimized out>)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Python/pythonrun.c:935
#9  0x00007f7f28b19f6c in sys_excepthook (
    self=self@entry=<module at remote 0x7f7f2828ef98>, 
    args=args@entry=(<type at remote 0x56025b2e1208>, <XMLSyntaxError(error_log=<lxml.etree._ListErrorLog at remote 0x7f7f279380e8>, code=1) at remote 0x7f7f27999d38>, <traceback at remote 0x7f7f27935688>))
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Python/sysmodule.c:321
#10 0x00007f7f28a24ef3 in _PyMethodDef_RawFastCallDict (
    method=<optimized out>, self=<optimized out>, args=<optimized out>, 
    nargs=<optimized out>, kwargs=<optimized out>)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Objects/call.c:518
#11 0x00007f7f28a25150 in _PyCFunction_FastCallDict (
    func=<built-in method excepthook of module object at remote 0x7f7f2828ef98>, args=<optimized out>, nargs=<optimized out>, kwargs=<optimized out>)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Objects/call.c:582
#12 0x00007f7f28b2d73e in PyErr_PrintEx (set_sys_last_vars=1)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Python/pythonrun.c:679
#13 PyErr_PrintEx (set_sys_last_vars=<optimized out>, 
    set_sys_last_vars=<optimized out>)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Python/pythonrun.c:641
#14 0x00007f7f289cd3a3 in PyRun_SimpleFileExFlags (fp=<optimized out>, 
    filename=<optimized out>, closeit=<optimized out>, flags=0x7ffd3b6ece90)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Python/pythonrun.c:435
#15 0x00007f7f28b2f7f1 in pymain_run_file (p_cf=0x7ffd3b6ece90, 
    filename=0x56025b210df0 L"sf.py", fp=0x56025b272ef0)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Modules/main.c:427
#16 pymain_run_filename (cf=0x7ffd3b6ece90, pymain=0x7ffd3b6ecfa0)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Modules/main.c:1627
#17 pymain_run_python (pymain=0x7ffd3b6ecfa0)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Modules/main.c:2876
#18 pymain_main (pymain=<optimized out>, pymain=<optimized out>)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Modules/main.c:3037
#19 0x00007f7f28b2fa2c in _Py_UnixMain (argc=<optimized out>, 
    argv=<optimized out>)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Modules/main.c:3072
#20 0x00007f7f2859d413 in __libc_start_main (main=0x56025a8fc050 <main>, 
    argc=2, argv=0x7ffd3b6ed0f8, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7ffd3b6ed0e8)
    at ../csu/libc-start.c:308
#21 0x000056025a8fc08e in _start ()

--- Additional comment from Jan Pazdziora on 2019-01-11 15:25:27 CET ---

Building python3-xmlsec from https://github.com/mehcode/python-xmlsec master (1.3.6) does not seem to improve the situation.

--- Additional comment from Jan Pazdziora on 2019-01-11 16:04:48 CET ---

Here by full, from different coredump though:

(gdb) bt full
#0  find_maxchar_surrogates (num_surrogates=<synthetic pointer>, maxchar=<synthetic pointer>, 
    end=0xfffffffffffffffd <error: Cannot access memory at address 0xfffffffffffffffd>, begin=0x1 <error: Cannot access memory at address 0x1>)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Objects/unicodeobject.c:1660
        iter = 0x1 <error: Cannot access memory at address 0x1>
        ch = <optimized out>
        iter = <optimized out>
        ch = <optimized out>
#1  _PyUnicode_Ready (unicode=b'sf.xml') at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Objects/unicodeobject.c:1699
        end = 0xfffffffffffffffd <error: Cannot access memory at address 0xfffffffffffffffd>
        maxchar = 0
        num_surrogates = 0
#2  0x00007fbac0b08651 in unicode_fromformat_write_str (precision=-1, width=-1, str=<optimized out>, writer=0x7ffd1669c160)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Objects/unicodeobject.c:2909
        length = <optimized out>
        fill = <optimized out>
        arglen = <optimized out>
        maxchar = <optimized out>
        length = <optimized out>
        fill = <optimized out>
        arglen = <optimized out>
        maxchar = <optimized out>
#3  unicode_fromformat_arg (vargs=0x7ffd1669c1c0, f=0x7fbac0cd7964 "U\", line %d\n", writer=0x7ffd1669c160)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Objects/unicodeobject.c:2801
        obj = b'sf.xml'
        len = <optimized out>
        zeropad = <optimized out>
        longflag = <optimized out>
        longlongflag = <optimized out>
        size_tflag = <optimized out>
        p = <optimized out>
        width = <optimized out>
        precision = -1
        fill = <optimized out>
        p = <optimized out>
        len = <optimized out>
        zeropad = <optimized out>
        width = <optimized out>
        precision = <optimized out>
        longflag = <optimized out>
        longlongflag = <optimized out>
        size_tflag = <optimized out>
        fill = <optimized out>
        ordinal = <optimized out>
        buffer = <optimized out>
        arglen = <optimized out>
        fillchar = <optimized out>
        number = <optimized out>
        s = <optimized out>
        obj = <optimized out>
        obj = <optimized out>
        str = <optimized out>
        obj = <optimized out>
        str = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        obj = <optimized out>
        repr = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        obj = <optimized out>
        ascii = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
#4  PyUnicode_FromFormatV (format=<optimized out>, vargs=<optimized out>)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Objects/unicodeobject.c:2908
        vargs2 = {{gp_offset = 16, fp_offset = 48, overflow_arg_area = 0x7ffd1669c300, reg_save_area = 0x7ffd1669c240}}
        f = <optimized out>
        writer = {
          buffer = '  File "\x01\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00w\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00ÿÿÿÿÿÿÿÿ\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00ÿ\x0f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', data = 0x7fbabfbbeab0, kind = PyUnicode_1BYTE_KIND, maxchar = 127, size = 121, pos = 8, min_length = 121, 
          min_char = 127, overallocate = 1 '\001', readonly = 0 '\000'}
#5  0x00007fbac0ba2d09 in PyUnicode_FromFormat (format=format@entry=0x7fbac0cd795b "  File \"%U\", line %d\n")
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Objects/unicodeobject.c:2960
        ret = <optimized out>
        vargs = {{gp_offset = 8, fp_offset = 48, overflow_arg_area = 0x7ffd1669c300, reg_save_area = 0x7ffd1669c240}}
#6  0x00007fbac0cba6f1 in print_exception (value=None, f=<_io.TextIOWrapper at remote 0x7fbac03ec708>)
    at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Python/pythonrun.c:753
        line = <optimized out>
        message = None
        filename = b'sf.xml'
        text = 0x0
        lineno = 0
        offset = -1
        err = <optimized out>
        type = <type at remote 0x560bb3d57208>
        tb = <traceback at remote 0x7fbabfae0608>
        err = <optimized out>
        type = <optimized out>
        tb = <optimized out>
        PyId_print_file_and_line = <optimized out>
        message = <optimized out>
        filename = <optimized out>
        text = <optimized out>
        lineno = <optimized out>
        offset = <optimized out>
        line = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--c
        moduleName = <optimized out>
        className = <optimized out>
        PyId___module__ = <optimized out>
        dot = <optimized out>
        _py_xdecref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        s = <optimized out>
        _py_xdecref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_xdecref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
#7  print_exception_recursive (f=<_io.TextIOWrapper at remote 0x7fbac03ec708>, value=<XMLSyntaxError(error_log=<lxml.etree._ListErrorLog at remote 0x7fbabfae11d8>, code=1) at remote 0x7fbabfb43d38>, seen=<optimized out>) at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Python/pythonrun.c:901
        err = 0
        res = <optimized out>
        cause = <optimized out>
        context = <optimized out>
#8  0x00007fbac0cbceeb in PyErr_Display (exception=<optimized out>, value=<XMLSyntaxError(error_log=<lxml.etree._ListErrorLog at remote 0x7fbabfae11d8>, code=1) at remote 0x7fbabfb43d38>, tb=<optimized out>) at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Python/pythonrun.c:935
        seen = {140440056905016}
        f = <_io.TextIOWrapper at remote 0x7fbac03ec708>
#9  0x00007fbac0cbcf6c in sys_excepthook (self=self@entry=<module at remote 0x7fbac0432f98>, args=args@entry=(<type at remote 0x560bb3d57208>, <XMLSyntaxError(error_log=<lxml.etree._ListErrorLog at remote 0x7fbabfae11d8>, code=1) at remote 0x7fbabfb43d38>, <traceback at remote 0x7fbabfae0608>)) at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Python/sysmodule.c:321
        exc = <type at remote 0x560bb3d57208>
        value = <XMLSyntaxError(error_log=<lxml.etree._ListErrorLog at remote 0x7fbabfae11d8>, code=1) at remote 0x7fbabfb43d38>
        tb = <traceback at remote 0x7fbabfae0608>
#10 0x00007fbac0bc7ef3 in _PyMethodDef_RawFastCallDict (method=<optimized out>, self=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwargs=<optimized out>) at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Objects/call.c:518
        argstuple = (<type at remote 0x560bb3d57208>, <XMLSyntaxError(error_log=<lxml.etree._ListErrorLog at remote 0x7fbabfae11d8>, code=1) at remote 0x7fbabfb43d38>, <traceback at remote 0x7fbabfae0608>)
        meth = <optimized out>
        flags = 1
        result = 0x0
#11 0x00007fbac0bc8150 in _PyCFunction_FastCallDict (func=<built-in method excepthook of module object at remote 0x7fbac0432f98>, args=<optimized out>, nargs=<optimized out>, kwargs=<optimized out>) at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Objects/call.c:582
        result = <optimized out>
#12 0x00007fbac0cd073e in PyErr_PrintEx (set_sys_last_vars=1) at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Python/pythonrun.c:679
        stack = {<type at remote 0x560bb3d57208>, <XMLSyntaxError(error_log=<lxml.etree._ListErrorLog at remote 0x7fbabfae11d8>, code=1) at remote 0x7fbabfb43d38>, <traceback at remote 0x7fbabfae0608>}
        result = <optimized out>
        exception = <type at remote 0x560bb3d57208>
        v = <XMLSyntaxError(error_log=<lxml.etree._ListErrorLog at remote 0x7fbabfae11d8>, code=1) at remote 0x7fbabfb43d38>
        tb = <traceback at remote 0x7fbabfae0608>
        hook = <optimized out>
        exception = <optimized out>
        v = <optimized out>
        tb = <optimized out>
        hook = <optimized out>
        stack = <optimized out>
        result = <optimized out>
        exception2 = <optimized out>
        v2 = <optimized out>
        tb2 = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_xdecref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_xdecref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_xdecref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_xdecref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_xdecref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
#13 PyErr_PrintEx (set_sys_last_vars=<optimized out>, set_sys_last_vars=<optimized out>) at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Python/pythonrun.c:641
        exception = <optimized out>
        v = <optimized out>
        tb = <optimized out>
        hook = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_xdecref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_xdecref_tmp = <optimized out>
        _py_xdecref_tmp = <optimized out>
        _py_xdecref_tmp = <optimized out>
        _py_xdecref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        exception2 = <optimized out>
        v2 = <optimized out>
        tb2 = <optimized out>
        stack = <optimized out>
        result = <optimized out>
        exception = <optimized out>
        v = <optimized out>
        tb = <optimized out>
        hook = <optimized out>
        _py_xdecref_tmp = <optimized out>
        _py_xdecref_tmp = <optimized out>
        _py_xdecref_tmp = <optimized out>
        _py_xdecref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        exception2 = <optimized out>
        v2 = <optimized out>
        tb2 = <optimized out>
        stack = <optimized out>
        result = <optimized out>
        _py_xdecref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_xdecref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_xdecref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
#14 0x00007fbac0b703a3 in PyRun_SimpleFileExFlags (fp=<optimized out>, filename=<optimized out>, closeit=<optimized out>, flags=0x7ffd1669c540) at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Python/pythonrun.c:435
        m = 0x0
        d = {'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='sf.py') at remote 0x7fbac03385c0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fbac0438c28>, '__file__': 'sf.py', '__cached__': None, 'etree': <module at remote 0x7fbac02b1688>, 'xmlsec': <module at remote 0x7fbabfad4228>, 'xml1': <lxml.etree._ElementTree at remote 0x7fbabfae0388>}
        v = <optimized out>
        ext = <optimized out>
        set_file_name = 1
        ret = -1
        len = <optimized out>
        _py_decref_tmp = <optimized out>
#15 0x00007fbac0cd27f1 in pymain_run_file (p_cf=0x7ffd1669c540, filename=0x560bb3c86df0 L"sf.py", fp=0x560bb3ce8ef0) at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Modules/main.c:427
        unicode = <optimized out>
        bytes = b'sf.py'
        filename_str = <optimized out>
        run = <optimized out>
        unicode = <optimized out>
        bytes = <optimized out>
        filename_str = <optimized out>
        run = <optimized out>
        _py_decref_tmp = <optimized out>
        _py_xdecref_tmp = <optimized out>
        _py_decref_tmp = <optimized out>
#16 pymain_run_filename (cf=0x7ffd1669c540, pymain=0x7ffd1669c650) at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Modules/main.c:1627
        fp = 0x560bb3ce8ef0
        fp = <optimized out>
#17 pymain_run_python (pymain=0x7ffd1669c650) at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Modules/main.c:2876
        cf = {cf_flags = 0}
        cf = <optimized out>
#18 pymain_main (pymain=<optimized out>, pymain=<optimized out>) at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Modules/main.c:3037
        res = <optimized out>
        res = <optimized out>
#19 0x00007fbac0cd2a2c in _Py_UnixMain (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Modules/main.c:3072
        pymain = {argc = 2, use_bytes_argv = 1, bytes_argv = 0x7ffd1669c7a8, wchar_argv = 0x0, status = 0, err = {prefix = 0x0, msg = 0x0, user_err = 0}, stdin_is_interactive = 1, skip_first_line = 0, filename = 0x560bb3c86df0 L"sf.py", command = 0x0, module = 0x0, main_importer_path = 0x0}
#20 0x00007fbac0740413 in __libc_start_main (main=0x560bb3778050 <main>, argc=2, argv=0x7ffd1669c7a8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd1669c798) at ../csu/libc-start.c:308
        self = <optimized out>
        result = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, -2611185483236398270, 94608255582304, 140724979484576, 0, 0, -8633824211270784190, -8601362210542343358}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x7ffd1669c7c0, 0x7fbac0e42150}, data = {prev = 0x0, cleanup = 0x0, canceltype = 376031168}}}
        not_first_call = <optimized out>
#21 0x0000560bb377808e in _start ()

--- Additional comment from Jan Pazdziora on 2019-01-11 16:15:10 CET ---

When I change the second etree.parse call to

xml2 = etree.parse("sf1.xml")

to try to open nonexistent file, I still get the segfault, even if I'd expect something like

<lxml.etree._ElementTree object at 0x7f29e4e65e08>
Traceback (most recent call last):
  File "sf.py", line 5, in <module>
    xml2 = etree.parse("sf1.xml")
  File "src/lxml/etree.pyx", line 3426, in lxml.etree.parse
  File "src/lxml/parser.pxi", line 1840, in lxml.etree._parseDocument
  File "src/lxml/parser.pxi", line 1866, in lxml.etree._parseDocumentFromURL
  File "src/lxml/parser.pxi", line 1770, in lxml.etree._parseDocFromFile
  File "src/lxml/parser.pxi", line 1163, in lxml.etree._BaseParser._parseDocFromFile
  File "src/lxml/parser.pxi", line 601, in lxml.etree._ParserContext._handleParseResultDoc
  File "src/lxml/parser.pxi", line 711, in lxml.etree._handleParseResult
  File "src/lxml/parser.pxi", line 638, in lxml.etree._raiseParseError
OSError: Error reading file 'sf1.xml': failed to load external entity "sf1.xml"

So one issue is why that second etree.parse on perfectly valid XML (as described in comment 0) fails and causes the exception.

But second issue is that when xmlsec is imported, python itself is not able to display that

OSError: Error reading file 'sf1.xml': failed to load external entity "sf1.xml"

message at all.

Comment 1 Jan Pazdziora 2019-01-11 15:20:54 UTC
Let's make this bugzilla the one for the problem of Objects/unicodeobject.c with code

int
_PyUnicode_Ready(PyObject *unicode)
{
    wchar_t *end;
    Py_UCS4 maxchar = 0;
    Py_ssize_t num_surrogates;
#if SIZEOF_WCHAR_T == 2
    Py_ssize_t length_wo_surrogates;
#endif

    /* _PyUnicode_Ready() is only intended for old-style API usage where
       strings were created using _PyObject_New() and where no canonical
       representation (the str field) has been set yet aka strings
       which are not yet ready. */
    assert(_PyUnicode_CHECK(unicode));
    assert(_PyUnicode_KIND(unicode) == PyUnicode_WCHAR_KIND);
    assert(_PyUnicode_WSTR(unicode) != NULL);
    assert(_PyUnicode_DATA_ANY(unicode) == NULL);
    assert(_PyUnicode_UTF8(unicode) == NULL);
    /* Actually, it should neither be interned nor be anything else: */
    assert(_PyUnicode_STATE(unicode).interned == SSTATE_NOT_INTERNED);

    end = _PyUnicode_WSTR(unicode) + _PyUnicode_WSTR_LENGTH(unicode);
    if (find_maxchar_surrogates(_PyUnicode_WSTR(unicode), end,
                                &maxchar, &num_surrogates) == -1)
        return -1;

turning that unicode parameter into something with end = 0xfffffffffffffffd on which find_maxchar_surrogates fails.

#1  _PyUnicode_Ready (unicode=b'sf.xml') at /usr/src/debug/python3-3.7.2-1.fc29.x86_64/Objects/unicodeobject.c:1699
        end = 0xfffffffffffffffd <error: Cannot access memory at address 0xfffffffffffffffd>
        maxchar = 0
        num_surrogates = 0

The original bug 1665459 would then be for the original issue -- why can't lxml load things twice.

Comment 2 Petr Viktorin (pviktori) 2019-01-11 15:39:22 UTC
#1  _PyUnicode_Ready (unicode=b'sg.xml')

Somehow, a bytes object, rather than a (Unicode) string, is used as the filename in a backtrace. That's an entirely different datatype at the C level.

I don't think Python would be setting b'sg.xml' as the filename of running code, but I don't know enough about lxml or xmlsec to tell why/where they would do it.

Comment 3 Petr Viktorin (pviktori) 2019-01-21 13:15:20 UTC
This is currently assigned to python-maint, and we have it in low priority. If you (anyone) can get to it faster than we can, please take the bug.

Comment 4 Victor Stinner 2019-07-01 10:37:03 UTC
I reported this bug upstream and I proposed a fix:
* https://bugs.python.org/issue37467
* https://github.com/python/cpython/pull/14504

Comment 5 Victor Stinner 2019-07-02 12:59:41 UTC
I consider that this low priority bug can wait for the next Python 3.7.4 release (scheduled for next days).

Comment 6 Fedora Update System 2019-07-09 17:26:43 UTC
FEDORA-2019-9bfb4a3e4b has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-9bfb4a3e4b

Comment 7 Fedora Update System 2019-07-09 17:29:55 UTC
FEDORA-2019-60a1defcd1 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-60a1defcd1

Comment 8 Fedora Update System 2019-07-10 00:37:42 UTC
python3-3.7.4-1.fc30, python3-docs-3.7.4-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-9bfb4a3e4b

Comment 9 Fedora Update System 2019-07-10 02:41:40 UTC
python3-3.7.4-1.fc29, python3-docs-3.7.4-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-60a1defcd1

Comment 10 Jan Pazdziora 2019-07-19 13:33:16 UTC
$ I don't see the issue fixed with python3-3.7.4-1.fc30.x86_64, I'm still getting

python3 sf.py 
<lxml.etree._ElementTree object at 0x7f80cc91d640>
Traceback (most recent call last):
  File "sf.py", line 5, in <module>
    xml2 = etree.parse("sf1.xml")
  File "src/lxml/etree.pyx", line 3426, in lxml.etree.parse
  File "src/lxml/parser.pxi", line 1840, in lxml.etree._parseDocument
  File "src/lxml/parser.pxi", line 1866, in lxml.etree._parseDocumentFromURL
  File "src/lxml/parser.pxi", line 1770, in lxml.etree._parseDocFromFile
  File "src/lxml/parser.pxi", line 1163, in lxml.etree._BaseParser._parseDocFromFile
  File "src/lxml/parser.pxi", line 601, in lxml.etree._ParserContext._handleParseResultDoc
  File "src/lxml/parser.pxi", line 711, in lxml.etree._handleParseResult
  File "src/lxml/parser.pxi", line 651, in lxml.etree._raiseParseError
Segmentation fault (core dumped)

Comment 11 Victor Stinner 2019-07-30 16:06:44 UTC
Sadly, Python 3.7 release manager decided to not pick my fix into 3.7.4 final release :-(

https://github.com/python/cpython/commit/8cbffc4d96d1da0fbc38da6f34f2da30c5ffd601

The fix will be part of next 3.7.5 release scheduled for the end of next September ("3.7.5 final: 2019-09-30 (expected)" says the PEP 537).

Comment 12 Petr Viktorin (pviktori) 2019-10-29 14:10:04 UTC
3.7.5 has been updated. Could you check if it fixes the problem?

Comment 13 Victor Stinner 2019-10-29 14:39:54 UTC
I tested the python37 in Rawhide (Python 3.7.5): the script doesn't crash anymore.

sudo dnf install -y xmlsec1 xmlsec1-devel libtool-ltdl-devel
python3.7 -m venv env
env/bin/python -m pip install lxml xmlsec
env/bin/python sf.py 

Output:

func=xmlSecCryptoDLLibraryCreate:file=dl.c:line=135:obj=unknown:subj=lt_dlopenext:error=7:io function failed:name="libxmlsec1-openssl"; errno=2
func=xmlSecCryptoDLGetLibraryFunctions:file=dl.c:line=442:obj=unknown:subj=xmlSecCryptoDLLibraryCreate:error=1:xmlsec library function failed:crypto=openssl
func=xmlSecCryptoDLLoadLibrary:file=dl.c:line=400:obj=unknown:subj=xmlSecCryptoDLGetLibraryFunctions:error=1:xmlsec library function failed: 
Traceback (most recent call last):
  File "sf.py", line 3, in <module>
    import xmlsec
SystemError: null argument to internal routine

Comment 14 Fedora Update System 2019-10-29 16:31:24 UTC
FEDORA-2019-232f092db0 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-232f092db0

Comment 15 Fedora Update System 2019-10-29 16:31:28 UTC
FEDORA-2019-aba3cca74a has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-aba3cca74a

Comment 16 Fedora Update System 2019-10-29 16:31:30 UTC
FEDORA-2019-986622833f has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-986622833f

Comment 17 Ben Cotton 2019-10-31 18:43:13 UTC
This message is a reminder that Fedora 29 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '29'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 29 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 18 Jan Pazdziora 2019-11-01 09:28:47 UTC
Confirming the fix in Fedora 29 (container) as well, with python3-3.7.5-1.fc29.x86_64.

Comment 19 Fedora Update System 2019-11-02 01:12:40 UTC
python3-3.7.5-1.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 20 Fedora Update System 2019-11-02 01:43:48 UTC
python3-3.7.5-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

Comment 21 Fedora Update System 2019-11-02 02:27:33 UTC
python3-3.7.5-1.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.

Comment 22 Fedora Update System 2019-11-03 00:11:13 UTC
python3-3.7.5-1.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.