Bug 1038813

+++ This bug was initially created as a clone of Bug #904996 +++

Description of problem:
By the a bit extensive interpretation of https://fedoraproject.org/wiki/Packaging:Guidelines#Test_Suites (“If the source code of the package provides a test suite, it should be executed in the %check section, whenever it is practical to do so.”) I would argue, that the testsuite shouldn't be packaged in RPM. But it should be run in %check.

That is obviously not the case with m2crypto (see attached results).

Version-Release number of selected component (if applicable):
m2crypto-0.21.1-9

How reproducible:
100%

Steps to Reproduce:
1.add %check section
2.
3.
  
Actual results:
a lot of fails, and what's even worse errors

Expected results:
clean run

Additional info:

--- Additional comment from Matěj Cepl on 2013-01-28 17:23:59 EST ---

I have created a special branch fedora_patches in my repo http://luther.ceplovi.cz/git/m2crypto.git/log/ and I have rebased python3 on it. This is the result of running ./fedora_setup.sh test in the top directory.

--- Additional comment from Miloslav Trmač on 2013-02-13 17:22:52 EST ---

I'm not sure what the log (containing no error messages) is supposed to indicate.

(We should definitely work on using a non-fixed port in the test suite, running it at %check, and not shipping it in the package.  However, are there any failures or errors or other non-packaging concerns in the current rawhide package that need solving?)

--- Additional comment from Matěj Cepl on 2013-03-07 11:11:42 EST ---

Yes, you are right, tests (or library) has been fixed in meanwhile. WIth the suggested patch all tests are satisfied and not installed.

--- Additional comment from Miloslav Trmač on 2013-03-11 12:00:52 EDT ---

This patch adds the necessary BuildRequires:

The resulting srpm builds fine in mock, but seems to hang in koji, last line is
> test_pass (tests.test_ssl.PassSSLClientTestCase) ... ok

I'll need to take a closer look.

--- Additional comment from Fedora End Of Life on 2013-04-03 12:21:05 EDT ---

This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.

(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19

--- Additional comment from Miloslav Trmač on 2013-11-22 16:30:34 EST ---

Also needs to modify alltests.py to read something like
>        r = unittest.TextTestRunner(verbosity=2).run(suite())
...
>    if not r.wasSuccessful():
>        import sys
>        sys.exit(1)

otherwise the test failure wouldn't be noticed by RPM.

--- Additional comment from Matěj Cepl on 2013-11-24 16:14:09 EST ---

Updated state of this bug is in http://pkgs.fedoraproject.org/cgit/m2crypto.git/log/?h=check-section-rhbz904996. Build still fails (.build*.log is also commited to that branch). We need to get some grip to the handling of missing EC modules.

Any idea, what kind of test we can design to test for say which tests are expected to fail?

======================================================================
ERROR: test_compute_key (tests.test_ecdh.ECDHTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/tests/test_ecdh.py", line 34, in test_compute_key
    a = EC.load_key(self.privkey)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 283, in load_key
    return load_key_bio(bio, callback)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 296, in load_key_bio
    return EC(m2.ec_key_read_bio(bio._ptr(), callback), 1)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 106, in __init__
    assert m2.ec_key_type_check(ec), "'ec' type error"
ValueError: Received a NULL pointer.

======================================================================
ERROR: test_pubkey_from_der (tests.test_ecdh.ECDHTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/tests/test_ecdh.py", line 42, in test_pubkey_from_der
    a = EC.gen_params(EC.NID_sect233k1)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 269, in gen_params
    return EC(m2.ec_key_new_by_curve_name(curve), 1)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 106, in __init__
    assert m2.ec_key_type_check(ec), "'ec' type error"
ValueError: Received a NULL pointer.

======================================================================
ERROR: test_genparam (tests.test_ecdsa.ECDSATestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/tests/test_ecdsa.py", line 75, in test_genparam
    ec = EC.gen_params(EC.NID_sect233k1)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 269, in gen_params
    return EC(m2.ec_key_new_by_curve_name(curve), 1)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 106, in __init__
    assert m2.ec_key_type_check(ec), "'ec' type error"
ValueError: Received a NULL pointer.

======================================================================
ERROR: test_loadkey (tests.test_ecdsa.ECDSATestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/tests/test_ecdsa.py", line 43, in test_loadkey
    ec = EC.load_key(self.privkey)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 283, in load_key
    return load_key_bio(bio, callback)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 296, in load_key_bio
    return EC(m2.ec_key_read_bio(bio._ptr(), callback), 1)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 106, in __init__
    assert m2.ec_key_type_check(ec), "'ec' type error"
ValueError: Received a NULL pointer.

======================================================================
ERROR: test_loadpubkey (tests.test_ecdsa.ECDSATestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/tests/test_ecdsa.py", line 48, in test_loadpubkey
    ec = EC.load_pub_key(self.pubkey)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 309, in load_pub_key
    return load_pub_key_bio(bio)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 325, in load_pub_key_bio
    ec_error()
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 329, in ec_error
    raise ECError, m2.err_reason_error_string(m2.err_get_error())
ECError: ASN1 lib

======================================================================
ERROR: test_sign_dsa_asn1 (tests.test_ecdsa.ECDSATestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/tests/test_ecdsa.py", line 62, in test_sign_dsa_asn1
    ec = EC.load_key(self.privkey)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 283, in load_key
    return load_key_bio(bio, callback)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 296, in load_key_bio
    return EC(m2.ec_key_read_bio(bio._ptr(), callback), 1)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 106, in __init__
    assert m2.ec_key_type_check(ec), "'ec' type error"
ValueError: Received a NULL pointer.

======================================================================
ERROR: test_verify_dsa (tests.test_ecdsa.ECDSATestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/tests/test_ecdsa.py", line 68, in test_verify_dsa
    ec = EC.load_key(self.privkey)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 283, in load_key
    return load_key_bio(bio, callback)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 296, in load_key_bio
    return EC(m2.ec_key_read_bio(bio._ptr(), callback), 1)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 106, in __init__
    assert m2.ec_key_type_check(ec), "'ec' type error"
ValueError: Received a NULL pointer.

======================================================================
ERROR: test_ec_curves_ECDSA (tests.test_ec_curves.ECCurveTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/tests/test_ec_curves.py", line 152, in test_ec_curves_ECDSA
    self.sign_verify_ecdsa(curveName, curveLen)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/tests/test_ec_curves.py", line 145, in sign_verify_ecdsa
    ec = self.genkey(curveName, curveLen)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/tests/test_ec_curves.py", line 131, in genkey
    ec = EC.gen_params(curve)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 269, in gen_params
    return EC(m2.ec_key_new_by_curve_name(curve), 1)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 106, in __init__
    assert m2.ec_key_type_check(ec), "'ec' type error"
ValueError: Received a NULL pointer.

----------------------------------------------------------------------
Ran 240 tests in 93.928s

FAILED (errors=8)
EC modules are available
EC modules are available
EC modules are available
Trying:
    check = Checker()
Expecting nothing
ok
Trying:
    check._match(host='my.example.com', certHost='my.example.com')
Expecting:
    True
ok
Trying:
    check._match(host='my.example.com', certHost='*.example.com')
Expecting:
    True
ok
Trying:
    check._match(host='my.example.com', certHost='m*.example.com')
Expecting:
    True
ok
Trying:
    check._match(host='my.example.com', certHost='m*.EXAMPLE.com')
Expecting:
    True
ok
Trying:
    check._match(host='my.example.com', certHost='m*ample.com')
Expecting:
    False
ok
Trying:
    check._match(host='my.example.com', certHost='*.*.com')
Expecting:
    False
ok
Trying:
    check._match(host='1.2.3.4', certHost='1.2.3.4')
Expecting:
    True
ok
Trying:
    check._match(host='1.2.3.4', certHost='*.2.3.4')
Expecting:
    False
ok
Trying:
    check._match(host='1234', certHost='1234')
Expecting:
    True
ok
Trying:
    check = Checker()
Expecting nothing
ok
Trying:
    check._splitSubjectAltName(host='my.example.com', subjectAltName='DNS:my.example.com')
Expecting:
    True
ok
Trying:
    check._splitSubjectAltName(host='my.example.com', subjectAltName='DNS:*.example.com')
Expecting:
    True
ok
Trying:
    check._splitSubjectAltName(host='my.example.com', subjectAltName='DNS:m*.example.com')
Expecting:
    True
ok
Trying:
    check._splitSubjectAltName(host='my.example.com', subjectAltName='DNS:m*ample.com')
Expecting:
    False
ok
Trying:
    check.useSubjectAltNameOnly
Expecting:
    True
ok
Trying:
    check._splitSubjectAltName(host='my.example.com', subjectAltName='DNS:m*ample.com, othername:<unsupported>')
Expecting:
    False
ok
Trying:
    check._splitSubjectAltName(host='my.example.com', subjectAltName='DNS:m*ample.com, DNS:my.example.org')
Expecting:
    False
ok
Trying:
    check._splitSubjectAltName(host='my.example.com', subjectAltName='DNS:m*ample.com, DNS:my.example.com')
Expecting:
    True
ok
Trying:
    check._splitSubjectAltName(host='my.example.com', subjectAltName='DNS:my.example.com, DNS:my.example.org')
Expecting:
    True
ok
Trying:
    check.useSubjectAltNameOnly
Expecting:
    True
ok
Trying:
    check._splitSubjectAltName(host='my.example.com', subjectAltName='')
Expecting:
    False
ok
Trying:
    check._splitSubjectAltName(host='my.example.com', subjectAltName='othername:<unsupported>')
Expecting:
    False
ok
Trying:
    check.useSubjectAltNameOnly
Expecting:
    False
ok
10 items had no tests:
    M2Crypto.SSL.Checker
    M2Crypto.SSL.Checker.Checker
    M2Crypto.SSL.Checker.Checker.__call__
    M2Crypto.SSL.Checker.Checker.__init__
    M2Crypto.SSL.Checker.NoCertificate
    M2Crypto.SSL.Checker.SSLVerificationError
    M2Crypto.SSL.Checker.WrongCertificate
    M2Crypto.SSL.Checker.WrongHost
    M2Crypto.SSL.Checker.WrongHost.__init__
    M2Crypto.SSL.Checker.WrongHost.__str__
2 items passed all tests:
  10 tests in M2Crypto.SSL.Checker.Checker._match
  14 tests in M2Crypto.SSL.Checker.Checker._splitSubjectAltName
24 tests in 12 items.
24 passed and 0 failed.
Test passed.
error: Bad exit status from /var/tmp/rpm-tmp.KnShOw (%check)

--- Additional comment from Miloslav Trmač on 2013-11-25 07:40:57 EST ---

(In reply to Matěj Cepl from comment #7)
> Updated state of this bug is in
> http://pkgs.fedoraproject.org/cgit/m2crypto.git/log/?h=check-section-
> rhbz904996. Build still fails (.build*.log is also commited to that branch).
> We need to get some grip to the handling of missing EC modules.

I haven't checked the details below yet - just a single comment from a quick glance: there already is a conditional for EC in alltests.py:
>     if m2.OPENSSL_VERSION_NUMBER >= 0x90800F and m2.OPENSSL_NO_EC == 0:
>         modules_to_test.append('tests.test_ecdh')
>         modules_to_test.append('tests.test_ecdsa')
>         modules_to_test.append('tests.test_ec_curves')
Why are the tests your patch adds necessary?