Bug 794950 - incoherent versioning of openssl breaks python selftest suite
Summary: incoherent versioning of openssl breaks python selftest suite
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: openssl
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Tomas Mraz
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 988757
TreeView+ depends on / blocked
 
Reported: 2012-02-18 03:40 UTC by Toshio Ernie Kuratomi
Modified: 2013-07-26 10:05 UTC (History)
7 users (show)

Fixed In Version: openssl-1.0.1-0.2.beta3.fc18
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 988757 (view as bug list)
Environment:
Last Closed: 2012-03-05 07:33:09 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Toshio Ernie Kuratomi 2012-02-18 03:40:11 UTC
Description of problem:

python is failing to build from source on F17 and rawhide.  Needed to add an Obsoletes for python-sqlite2 and that's how I discovered this.

Version-Release number of selected component (if applicable):
python-2.7.2-19.fc18

How reproducible:
Everytime

Steps to Reproduce:
1. fedpkg clone python
2. cd python
3. fedpkg build
  
Actual results:

Build fails

Expected results:

Build succeeds

Additional info:

F17: http://koji.fedoraproject.org/koji/taskinfo?taskID=3800336
F18: http://koji.fedoraproject.org/koji/taskinfo?taskID=3800334

Same problems with the tests:

347 tests OK.
1 test failed:
    test_ssl
39 tests skipped:
    test_aepack test_al test_applesingle test_bsddb185 test_bsddb3
    test_cd test_cl test_codecmaps_cn test_codecmaps_hk
    test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
    test_dl test_gdb test_gl test_imgfile test_ioctl test_kqueue
    test_linuxaudiodev test_macos test_macostools test_msilib
    test_ossaudiodev test_pep277 test_scriptpackages test_smtpnet
    test_socketserver test_startfile test_sunaudiodev test_timeout
    test_tk test_ttk_guionly test_unicode_file test_urllib2net
    test_urllibnet test_winreg test_winsound test_zipfile64
4 skips unexpected on linux2:
    test_gdb test_ioctl test_tk test_ttk_guionly
[1219652 refs]
make: *** [test] Error 1

Comment 1 Dave Malcolm 2012-02-18 21:03:19 UTC
Both F17 and F18 builds failed due to this test case failing within test_ssl:
======================================================================
FAIL: test_openssl_version (test.test_ssl.BasicSocketTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Python-2.7.2/Lib/test/test_ssl.py", line 151, in test_openssl_version
    (s, t))
AssertionError: ('OpenSSL 1.0.1-fips-beta2 19 Jan 2012', (1, 0, 0, 0, 3))
----------------------------------------------------------------------

Comment 2 Dave Malcolm 2012-02-18 21:13:49 UTC
Both the f17 and f18 builds were against openssl-1.0.1-0.1.beta2.fc17

The last successful build of python was python-2.7.2-18.fc17:
  http://koji.fedoraproject.org/koji/buildinfo?buildID=281149
which was against openssl-1.0.0e-3.fc17

The failing assertion is (edited for conciseness):

        s = ssl.OPENSSL_VERSION
        t = ssl.OPENSSL_VERSION_INFO
        major, minor, fix, patch, status = t
        # Version string as returned by OpenSSL, the format might change
        self.assertTrue(s.startswith("OpenSSL {:d}.{:d}.{:d}".format(major, minor, fix)),
                        (s, t))

It looks like:

  ssl.OPENSSL_VERSION == 'OpenSSL 1.0.1-fips-beta2 19 Jan 2012'
  ssl.OPENSSL_VERSION_INFO == (1, 0, 0, 0, 3)

and the assertion is complaining because the string version starts with 'OpenSSL 1.0.1', whereas the tuple version is somehow 1.0.0

This probably affects the "python3" src.rpm also: the test case in Python-3.2.2 appears to share the same code.

Comment 3 Dave Malcolm 2012-02-18 21:19:59 UTC
Looking in Python-2.7.2/Modules/_ssl.c:

 ssl.OPENSSL_VERSION comes from:
    SSLeay_version(SSLEAY_VERSION)
 via this code:
  1748      r = PyString_FromString(SSLeay_version(SSLEAY_VERSION));
  1749      if (r == NULL || PyModule_AddObject(m, "OPENSSL_VERSION", r))
  1750          return;

 ssl.OPENSSL_VERSION_INFO comes from:
     SSLeay()
 via this code:
  1726      /* OpenSSL version */
  1727      /* SSLeay() gives us the version of the library linked against,
  1728         which could be different from the headers version.
  1729      */
  1730      libver = SSLeay();
  1731      r = PyLong_FromUnsignedLong(libver);
  1732      if (r == NULL)
  1733          return;
  1734      if (PyModule_AddObject(m, "OPENSSL_VERSION_NUMBER", r))
  1735          return;
  1736      status = libver & 0xF;
  1737      libver >>= 4;
  1738      patch = libver & 0xFF;
  1739      libver >>= 8;
  1740      fix = libver & 0xFF;
  1741      libver >>= 8;
  1742      minor = libver & 0xFF;
  1743      libver >>= 8;
  1744      major = libver & 0xFF;
  1745      r = Py_BuildValue("IIIII", major, minor, fix, patch, status);
  1746      if (r == NULL || PyModule_AddObject(m, "OPENSSL_VERSION_INFO", r))
  1747          return;

Comment 4 Dave Malcolm 2012-02-18 21:22:00 UTC
CCing tmraz (openssl owner)

tmraz: any thoughts?  are SSLeay() and SSLeay_version(SSLEAY_VERSION) out-of-sync somehow?

Comment 5 Dave Malcolm 2012-02-22 01:50:24 UTC
In /usr/src/debug/openssl-1.0.1-beta2/crypto/cversion.c:
    65  const char *SSLeay_version(int t)
    66          {
    67          if (t == SSLEAY_VERSION)
    68                  return OPENSSL_VERSION_TEXT;

and:
   113  unsigned long SSLeay(void)
   114          {
   115          return(SSLEAY_VERSION_NUMBER);
   116          }

/usr/include/openssl/opensslv.h has:
    28  #define OPENSSL_VERSION_NUMBER  0x10000003L
    29  #ifdef OPENSSL_FIPS
    30  #define OPENSSL_VERSION_TEXT    "OpenSSL 1.0.1-fips-beta2 19 Jan 2012"
    31  #else
    32  #define OPENSSL_VERSION_TEXT    "OpenSSL 1.0.1-beta2 19 Jan 2012"
    33  #endif

Reassigning to openssl

Comment 6 Dave Malcolm 2012-02-22 01:51:17 UTC
This is with openssl-1.0.1-0.1.beta2.fc17.x86_64

Comment 7 Vít Ondruch 2012-02-24 10:35:21 UTC
Also Ruby are FTBFS. I cannot build updated Ruby. The following tests are failing:

https://github.com/ruby/ruby/blob/trunk/test/openssl/test_x509cert.rb
https://github.com/ruby/ruby/blob/trunk/test/drb/test_drbssl.rb

Comment 8 Dave Malcolm 2012-02-24 16:14:53 UTC
(In reply to comment #7)
> Also Ruby are FTBFS. I cannot build updated Ruby. The following tests are
> failing:
> 
> https://github.com/ruby/ruby/blob/trunk/test/openssl/test_x509cert.rb
> https://github.com/ruby/ruby/blob/trunk/test/drb/test_drbssl.rb

For reference, which of these tests are failing, and how?

Comment 9 Vít Ondruch 2012-02-24 16:42:55 UTC
(In reply to comment #8)
> (In reply to comment #7)
> > Also Ruby are FTBFS. I cannot build updated Ruby. The following tests are
> > failing:
> > 
> > https://github.com/ruby/ruby/blob/trunk/test/openssl/test_x509cert.rb
> > https://github.com/ruby/ruby/blob/trunk/test/drb/test_drbssl.rb
> 
> For reference, which of these tests are failing, and how?

Actually the failures are bit different than what you described here. I opened separate issue for Ruby: https://bugzilla.redhat.com/show_bug.cgi?id=797217

Comment 10 Dave Malcolm 2012-02-24 16:48:55 UTC
Thanks.  Retitling this issue to restrict it to the thing that breaks the Python test (incoherent versioning).

Comment 11 Tomas Mraz 2012-02-27 08:51:09 UTC
Can you please drop this test from the python testsuite? Unfortunately the numeric version number has to be kept the same as was in the 1.0.0 version because it is part of the ABI. I've never seen any other application that would compare the numeric version number with the string.

Comment 12 Tomas Mraz 2012-02-28 21:04:40 UTC
I'll try to provide symbol versioning for the SSLeay() function so it returns the old version to apps compiled against 1.0.0 and a new version against apps built with 1.0.1.

Comment 13 Tomas Mraz 2012-02-29 21:18:58 UTC
Please try the build against the current rawhide - it should now return coherent version for newly built apps.

Comment 14 Thomas Spura 2012-03-01 15:29:30 UTC
Works:
http://koji.fedoraproject.org/koji/taskinfo?taskID=3843404

test_ssl
test_sslwrap_simple (test.test_ssl.BasicTests) ... ok
test_DER_to_PEM (test.test_ssl.BasicSocketTests) ... ok
test_ciphers (test.test_ssl.BasicSocketTests) ... ok
test_constants (test.test_ssl.BasicSocketTests) ... ok
test_openssl_version (test.test_ssl.BasicSocketTests) ... ok
test_parse_cert (test.test_ssl.BasicSocketTests) ... 
{'notAfter': 'Feb 16 16:54:50 2013 GMT',
 'subject': ((('countryName', u'US'),),
             (('stateOrProvinceName', u'Delaware'),),
             (('localityName', u'Wilmington'),),
             (('organizationName', u'Python Software Foundation'),),
             (('organizationalUnitName', u'SSL'),),
             (('commonName', u'somemachine.python.org'),))}
ok

Could you do an update for f17 too please?

Comment 15 Tomas Mraz 2012-03-01 16:03:51 UTC
F17 is reverted to 1.0.0g due to other regressions.

See:
https://admin.fedoraproject.org/updates/FEDORA-2012-2665/openssl-1.0.0g-4.fc17

Comment 16 Henrik Nordström 2012-03-04 13:34:06 UTC
This fake OPENSSL_VERSION downgrade also bites Squid-3 on F16 which uses OPENSSL_VERSION to adjust to slight differences in a couple of openssl-1.0.x versions.

Comment 17 Tomas Mraz 2012-03-05 07:33:09 UTC
Unfortunately as I said there is no way to avoid that fake downgrade for released distros. On Fedora rawhide this is solved by using symbol versioning.


Note You need to log in before you can comment on or make changes to this bug.