Bug 904996

Summary: Run test suite in %check
Product: [Fedora] Fedora Reporter: Matěj Cepl <mcepl>
Component: m2cryptoAssignee: Miloslav Trmač <mitr>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: low Docs Contact:
Priority: unspecified    
Version: rawhideCC: gholms, mcepl, mitr
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: m2crypto-0.21.1-13.fc20 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1038813 (view as bug list) Environment:
Last Closed: 2014-01-03 08:42:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1038813    
Attachments:
Description Flags
result of the fedpkg local run
none
result of ./fedora_setup.sh test
none
suggested patch
none
More complete patch none

Description Matěj Cepl 2013-01-28 11:06:40 UTC
Created attachment 688873 [details]
result of the fedpkg local run

Description of problem:
By the a bit extensive interpretation of https://fedoraproject.org/wiki/Packaging:Guidelines#Test_Suites (“If the source code of the package provides a test suite, it should be executed in the %check section, whenever it is practical to do so.”) I would argue, that the testsuite shouldn't be packaged in RPM. But it should be run in %check.

That is obviously not the case with m2crypto (see attached results).

Version-Release number of selected component (if applicable):
m2crypto-0.21.1-9

How reproducible:
100%

Steps to Reproduce:
1.add %check section
2.
3.
  
Actual results:
a lot of fails, and what's even worse errors

Expected results:
clean run

Additional info:

Comment 1 Matěj Cepl 2013-01-28 22:23:59 UTC
Created attachment 689389 [details]
result of ./fedora_setup.sh test

I have created a special branch fedora_patches in my repo http://luther.ceplovi.cz/git/m2crypto.git/log/ and I have rebased python3 on it. This is the result of running ./fedora_setup.sh test in the top directory.

Comment 2 Miloslav Trmač 2013-02-13 22:22:52 UTC
I'm not sure what the log (containing no error messages) is supposed to indicate.

(We should definitely work on using a non-fixed port in the test suite, running it at %check, and not shipping it in the package.  However, are there any failures or errors or other non-packaging concerns in the current rawhide package that need solving?)

Comment 3 Matěj Cepl 2013-03-07 16:11:42 UTC
Created attachment 706713 [details]
suggested patch

Yes, you are right, tests (or library) has been fixed in meanwhile. WIth the suggested patch all tests are satisfied and not installed.

Comment 4 Miloslav Trmač 2013-03-11 16:00:52 UTC
Created attachment 708485 [details]
More complete patch

This patch adds the necessary BuildRequires:

The resulting srpm builds fine in mock, but seems to hang in koji, last line is
> test_pass (tests.test_ssl.PassSSLClientTestCase) ... ok

I'll need to take a closer look.

Comment 5 Fedora End Of Life 2013-04-03 16:21:05 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.

(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19

Comment 6 Miloslav Trmač 2013-11-22 21:30:34 UTC
Also needs to modify alltests.py to read something like
>        r = unittest.TextTestRunner(verbosity=2).run(suite())
...
>    if not r.wasSuccessful():
>        import sys
>        sys.exit(1)

otherwise the test failure wouldn't be noticed by RPM.

Comment 7 Matěj Cepl 2013-11-24 21:14:09 UTC
Updated state of this bug is in http://pkgs.fedoraproject.org/cgit/m2crypto.git/log/?h=check-section-rhbz904996. Build still fails (.build*.log is also commited to that branch). We need to get some grip to the handling of missing EC modules.

Any idea, what kind of test we can design to test for say which tests are expected to fail?

======================================================================
ERROR: test_compute_key (tests.test_ecdh.ECDHTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/tests/test_ecdh.py", line 34, in test_compute_key
    a = EC.load_key(self.privkey)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 283, in load_key
    return load_key_bio(bio, callback)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 296, in load_key_bio
    return EC(m2.ec_key_read_bio(bio._ptr(), callback), 1)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 106, in __init__
    assert m2.ec_key_type_check(ec), "'ec' type error"
ValueError: Received a NULL pointer.

======================================================================
ERROR: test_pubkey_from_der (tests.test_ecdh.ECDHTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/tests/test_ecdh.py", line 42, in test_pubkey_from_der
    a = EC.gen_params(EC.NID_sect233k1)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 269, in gen_params
    return EC(m2.ec_key_new_by_curve_name(curve), 1)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 106, in __init__
    assert m2.ec_key_type_check(ec), "'ec' type error"
ValueError: Received a NULL pointer.

======================================================================
ERROR: test_genparam (tests.test_ecdsa.ECDSATestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/tests/test_ecdsa.py", line 75, in test_genparam
    ec = EC.gen_params(EC.NID_sect233k1)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 269, in gen_params
    return EC(m2.ec_key_new_by_curve_name(curve), 1)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 106, in __init__
    assert m2.ec_key_type_check(ec), "'ec' type error"
ValueError: Received a NULL pointer.

======================================================================
ERROR: test_loadkey (tests.test_ecdsa.ECDSATestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/tests/test_ecdsa.py", line 43, in test_loadkey
    ec = EC.load_key(self.privkey)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 283, in load_key
    return load_key_bio(bio, callback)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 296, in load_key_bio
    return EC(m2.ec_key_read_bio(bio._ptr(), callback), 1)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 106, in __init__
    assert m2.ec_key_type_check(ec), "'ec' type error"
ValueError: Received a NULL pointer.

======================================================================
ERROR: test_loadpubkey (tests.test_ecdsa.ECDSATestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/tests/test_ecdsa.py", line 48, in test_loadpubkey
    ec = EC.load_pub_key(self.pubkey)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 309, in load_pub_key
    return load_pub_key_bio(bio)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 325, in load_pub_key_bio
    ec_error()
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 329, in ec_error
    raise ECError, m2.err_reason_error_string(m2.err_get_error())
ECError: ASN1 lib

======================================================================
ERROR: test_sign_dsa_asn1 (tests.test_ecdsa.ECDSATestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/tests/test_ecdsa.py", line 62, in test_sign_dsa_asn1
    ec = EC.load_key(self.privkey)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 283, in load_key
    return load_key_bio(bio, callback)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 296, in load_key_bio
    return EC(m2.ec_key_read_bio(bio._ptr(), callback), 1)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 106, in __init__
    assert m2.ec_key_type_check(ec), "'ec' type error"
ValueError: Received a NULL pointer.

======================================================================
ERROR: test_verify_dsa (tests.test_ecdsa.ECDSATestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/tests/test_ecdsa.py", line 68, in test_verify_dsa
    ec = EC.load_key(self.privkey)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 283, in load_key
    return load_key_bio(bio, callback)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 296, in load_key_bio
    return EC(m2.ec_key_read_bio(bio._ptr(), callback), 1)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 106, in __init__
    assert m2.ec_key_type_check(ec), "'ec' type error"
ValueError: Received a NULL pointer.

======================================================================
ERROR: test_ec_curves_ECDSA (tests.test_ec_curves.ECCurveTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/tests/test_ec_curves.py", line 152, in test_ec_curves_ECDSA
    self.sign_verify_ecdsa(curveName, curveLen)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/tests/test_ec_curves.py", line 145, in sign_verify_ecdsa
    ec = self.genkey(curveName, curveLen)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/tests/test_ec_curves.py", line 131, in genkey
    ec = EC.gen_params(curve)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 269, in gen_params
    return EC(m2.ec_key_new_by_curve_name(curve), 1)
  File "/home/matej/build/Extras/m2crypto/M2Crypto-0.21.1/M2Crypto/EC.py", line 106, in __init__
    assert m2.ec_key_type_check(ec), "'ec' type error"
ValueError: Received a NULL pointer.

----------------------------------------------------------------------
Ran 240 tests in 93.928s

FAILED (errors=8)
EC modules are available
EC modules are available
EC modules are available
Trying:
    check = Checker()
Expecting nothing
ok
Trying:
    check._match(host='my.example.com', certHost='my.example.com')
Expecting:
    True
ok
Trying:
    check._match(host='my.example.com', certHost='*.example.com')
Expecting:
    True
ok
Trying:
    check._match(host='my.example.com', certHost='m*.example.com')
Expecting:
    True
ok
Trying:
    check._match(host='my.example.com', certHost='m*.EXAMPLE.com')
Expecting:
    True
ok
Trying:
    check._match(host='my.example.com', certHost='m*ample.com')
Expecting:
    False
ok
Trying:
    check._match(host='my.example.com', certHost='*.*.com')
Expecting:
    False
ok
Trying:
    check._match(host='1.2.3.4', certHost='1.2.3.4')
Expecting:
    True
ok
Trying:
    check._match(host='1.2.3.4', certHost='*.2.3.4')
Expecting:
    False
ok
Trying:
    check._match(host='1234', certHost='1234')
Expecting:
    True
ok
Trying:
    check = Checker()
Expecting nothing
ok
Trying:
    check._splitSubjectAltName(host='my.example.com', subjectAltName='DNS:my.example.com')
Expecting:
    True
ok
Trying:
    check._splitSubjectAltName(host='my.example.com', subjectAltName='DNS:*.example.com')
Expecting:
    True
ok
Trying:
    check._splitSubjectAltName(host='my.example.com', subjectAltName='DNS:m*.example.com')
Expecting:
    True
ok
Trying:
    check._splitSubjectAltName(host='my.example.com', subjectAltName='DNS:m*ample.com')
Expecting:
    False
ok
Trying:
    check.useSubjectAltNameOnly
Expecting:
    True
ok
Trying:
    check._splitSubjectAltName(host='my.example.com', subjectAltName='DNS:m*ample.com, othername:<unsupported>')
Expecting:
    False
ok
Trying:
    check._splitSubjectAltName(host='my.example.com', subjectAltName='DNS:m*ample.com, DNS:my.example.org')
Expecting:
    False
ok
Trying:
    check._splitSubjectAltName(host='my.example.com', subjectAltName='DNS:m*ample.com, DNS:my.example.com')
Expecting:
    True
ok
Trying:
    check._splitSubjectAltName(host='my.example.com', subjectAltName='DNS:my.example.com, DNS:my.example.org')
Expecting:
    True
ok
Trying:
    check.useSubjectAltNameOnly
Expecting:
    True
ok
Trying:
    check._splitSubjectAltName(host='my.example.com', subjectAltName='')
Expecting:
    False
ok
Trying:
    check._splitSubjectAltName(host='my.example.com', subjectAltName='othername:<unsupported>')
Expecting:
    False
ok
Trying:
    check.useSubjectAltNameOnly
Expecting:
    False
ok
10 items had no tests:
    M2Crypto.SSL.Checker
    M2Crypto.SSL.Checker.Checker
    M2Crypto.SSL.Checker.Checker.__call__
    M2Crypto.SSL.Checker.Checker.__init__
    M2Crypto.SSL.Checker.NoCertificate
    M2Crypto.SSL.Checker.SSLVerificationError
    M2Crypto.SSL.Checker.WrongCertificate
    M2Crypto.SSL.Checker.WrongHost
    M2Crypto.SSL.Checker.WrongHost.__init__
    M2Crypto.SSL.Checker.WrongHost.__str__
2 items passed all tests:
  10 tests in M2Crypto.SSL.Checker.Checker._match
  14 tests in M2Crypto.SSL.Checker.Checker._splitSubjectAltName
24 tests in 12 items.
24 passed and 0 failed.
Test passed.
error: Bad exit status from /var/tmp/rpm-tmp.KnShOw (%check)

Comment 8 Miloslav Trmač 2013-11-25 12:40:57 UTC
(In reply to Matěj Cepl from comment #7)
> Updated state of this bug is in
> http://pkgs.fedoraproject.org/cgit/m2crypto.git/log/?h=check-section-
> rhbz904996. Build still fails (.build*.log is also commited to that branch).
> We need to get some grip to the handling of missing EC modules.

I haven't checked the details below yet - just a single comment from a quick glance: there already is a conditional for EC in alltests.py:
>     if m2.OPENSSL_VERSION_NUMBER >= 0x90800F and m2.OPENSSL_NO_EC == 0:
>         modules_to_test.append('tests.test_ecdh')
>         modules_to_test.append('tests.test_ecdsa')
>         modules_to_test.append('tests.test_ec_curves')
Why are the tests your patch adds necessary?

Comment 9 Miloslav Trmač 2013-12-18 00:20:57 UTC
(In reply to Miloslav Trmač from comment #6)
> Also needs to modify alltests.py to read something like
> >        r = unittest.TextTestRunner(verbosity=2).run(suite())
> ...
> >    if not r.wasSuccessful():
> >        import sys
> >        sys.exit(1)
> 
> otherwise the test failure wouldn't be noticed by RPM.

Actually only necessary for running tests/alltests.py directly, not for (python setup.py check).  Filed as https://bugzilla.osafoundation.org/show_bug.cgi?id=13099 but otherwise unnecessary.

Comment 10 Miloslav Trmač 2013-12-18 02:38:00 UTC
Fix pushed to rawhide, will build in F20 as well.  Can I delete the check-section-rhbz904996 branch?

Comment 11 Fedora Update System 2013-12-18 17:11:10 UTC
m2crypto-0.21.1-13.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/m2crypto-0.21.1-13.fc20

Comment 12 Fedora Update System 2013-12-20 01:53:44 UTC
Package m2crypto-0.21.1-13.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing m2crypto-0.21.1-13.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-23645/m2crypto-0.21.1-13.fc20
then log in and leave karma (feedback).

Comment 13 Fedora Update System 2014-01-03 08:42:27 UTC
m2crypto-0.21.1-13.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.