Bug 1457315

Summary: cmocka tests are not executed during rpm build
Product: Red Hat Enterprise Linux 7 Reporter: Viktor Ashirov <vashirov>
Component: 389-ds-baseAssignee: mreynolds
Status: CLOSED ERRATA QA Contact: Viktor Ashirov <vashirov>
Severity: unspecified Docs Contact:
Priority: medium    
Version: 7.4CC: arubin, mreynolds, nkinder, ovasik, rmeggins
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 389-ds-base-1.3.7.5-12 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-10 14:16:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
log for build failures with cmokca tests none

Description Viktor Ashirov 2017-05-31 13:39:21 UTC
Description of problem:
389-ds-base src.rpm requires for build libcmocka-devel:

$ rhpkg clone -a 389-ds-base
$ grep cmocka 389-ds-base/*.spec
BuildRequires:    libcmocka-devel

But rpm doesn't have %check:
$ grep \%check 389-ds-base/*.spec -c
0

So they're never actually executed during rpm build.

And if I add %check, it fails because not all files are present for tests to compile successfully:

[root@rhel7ds 389-ds-base-1.3.6.1]# make check 
make  check-am
make[1]: Entering directory `/root/rpmbuild/BUILD/389-ds-base-1.3.6.1'
make  test_slapd test_libsds benchmark_sds benchmark_par_sds test_nuncstans test_nuncstans_stress
make[2]: Entering directory `/root/rpmbuild/BUILD/389-ds-base-1.3.6.1'
make[2]: *** No rule to make target `test/libslapd/counters/atomic.c', needed by `test/libslapd/counters/test_slapd-atomic.o'.  Stop.
make[2]: Leaving directory `/root/rpmbuild/BUILD/389-ds-base-1.3.6.1'
make[1]: *** [check-am] Error 2
make[1]: Leaving directory `/root/rpmbuild/BUILD/389-ds-base-1.3.6.1'
make: *** [check] Error 2

This file landed to the tree in https://pagure.io/389-ds-base/c/15ce6462e619091897ecdb4497142ebf06835e55

Version-Release number of selected component (if applicable):
389-ds-base-1.3.6.1-15.el7.x86_64

Comment 2 wibrown@redhat.com 2017-06-05 22:39:18 UTC
Pretty sure that cmocka comes from EPEL?

Comment 3 mreynolds 2017-06-06 15:12:18 UTC
To get this to work we need to bring in some really large patches that do not apply to 7.4.  This should be pushed to 7.5

Comment 9 mreynolds 2017-06-06 19:58:35 UTC
Actually this is causing all kinds of build problems.  Please confirm if just adding the atomic.c file fixes things.  I could do a one off patch that just adds this file, but trying to back port these other patches is not working.

Comment 12 mreynolds 2017-10-18 17:43:17 UTC
We get failures when we enable cmocka tests:

make  check-TESTS
make[2]: Entering directory `/builddir/build/BUILD/389-ds-base-1.3.7.5'
make[3]: Entering directory `/builddir/build/BUILD/389-ds-base-1.3.7.5'
./test-driver: line 95: 167774 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: test_slapd
PASS: test_libsds
PASS: test_nuncstans
PASS: test_nuncstans_stress_small
make[4]: Entering directory `/builddir/build/BUILD/389-ds-base-1.3.7.5'
make  all-am
make[5]: Entering directory `/builddir/build/BUILD/389-ds-base-1.3.7.5'
make[5]: Leaving directory `/builddir/build/BUILD/389-ds-base-1.3.7.5'
make[4]: Leaving directory `/builddir/build/BUILD/389-ds-base-1.3.7.5'
============================================================================
Testsuite summary for 389-ds-base 1.3.7.5
============================================================================
# TOTAL: 4
# PASS:  3
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See ./test-suite.log
Please report to http://bugzilla.redhat.com/enter_bug.cgi?product=389
============================================================================
make[3]: *** [test-suite.log] Error 1
make[3]: Leaving directory `/builddir/build/BUILD/389-ds-base-1.3.7.5'
make[2]: *** [check-TESTS] Error 2
make[2]: Leaving directory `/builddir/build/BUILD/389-ds-base-1.3.7.5'
make[1]: *** [check-am] Error 2
make[1]: Leaving directory `/builddir/build/BUILD/389-ds-base-1.3.7.5'
make: *** [check] Error 2
+ cat ./test-suite.log
===========================================
   389-ds-base 1.3.7.5: ./test-suite.log
===========================================
# TOTAL: 4
# PASS:  3
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
.. contents:: :depth: 2
FAIL: test_slapd
================
[==========] Running 11 test(s).
[ RUN      ] test_libslapd_hello
[       OK ] test_libslapd_hello
[ RUN      ] test_libslapd_pblock_analytics
[       OK ] test_libslapd_pblock_analytics
[ RUN      ] test_libslapd_pblock_v3c_target_dn
[       OK ] test_libslapd_pblock_v3c_target_dn
[ RUN      ] test_libslapd_pblock_v3c_target_sdn
[       OK ] test_libslapd_pblock_v3c_target_sdn
[ RUN      ] test_libslapd_pblock_v3c_original_target_dn
[       OK ] test_libslapd_pblock_v3c_original_target_dn
[ RUN      ] test_libslapd_pblock_v3c_target_uniqueid
[       OK ] test_libslapd_pblock_v3c_target_uniqueid
[ RUN      ] test_libslapd_operation_v3c_target_spec
[       OK ] test_libslapd_operation_v3c_target_spec
[ RUN      ] test_libslapd_counters_atomic_usage
[       OK ] test_libslapd_counters_atomic_usage
[ RUN      ] test_libslapd_counters_atomic_overflow
[       OK ] test_libslapd_counters_atomic_overflow
[ RUN      ] test_libslapd_pal_meminfo
Test failed with exception: Segmentation fault
[  FAILED  ] test_libslapd_pal_meminfo
[ RUN      ] test_libslapd_util_cachesane

Comment 13 Viktor Ashirov 2017-10-18 18:14:09 UTC
I saw the same segfault once, but couldn't reproduce anymore: https://pagure.io/389-ds-base/issue/49384

Comment 14 Viktor Ashirov 2017-10-20 10:07:56 UTC
Mark,

I believe I found the root cause for this segfault. Brew build hosts have rather old kernel (2.6.32 on RHEL6), that doesn't have MemAvailable in /proc/meminfo. William recently fixed it in https://pagure.io/389-ds-base/issue/49392, but the change was not picked up by downstream build (BTW, we still have time to do another rebase by Monday). 

I did a couple of scratch builds with the patch applied, they completed successfully, except on ppc64, that has another (BE vs LE?) issue:

============================================================================
Testsuite summary for 389-ds-base 1.3.7.5
============================================================================
# TOTAL: 4
# PASS:  3
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See ./test-suite.log
Please report to http://bugzilla.redhat.com/enter_bug.cgi?product=389
============================================================================
make[3]: *** [test-suite.log] Error 1
make[3]: Leaving directory `/builddir/build/BUILD/389-ds-base-1.3.7.5'
make[2]: *** [check-TESTS] Error 2
make[2]: Leaving directory `/builddir/build/BUILD/389-ds-base-1.3.7.5'
make[1]: *** [check-am] Error 2
make[1]: Leaving directory `/builddir/build/BUILD/389-ds-base-1.3.7.5'
make: *** [check] Error 2
+ cat ./test-suite.log
===========================================
   389-ds-base 1.3.7.5: ./test-suite.log
===========================================
# TOTAL: 4
# PASS:  3
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
.. contents:: :depth: 2
FAIL: test_libsds
=================
[==========] Running 26 test(s).
[ RUN      ] test_1_invalid_binst_ptr
[       OK ] test_1_invalid_binst_ptr
[ RUN      ] test_3_single_insert
[       OK ] test_3_single_insert
[ RUN      ] test_4_single_null_insert_fn
[       OK ] test_4_single_null_insert_fn
[ RUN      ] test_6_insert_less_than_no_split
[       OK ] test_6_insert_less_than_no_split
[ RUN      ] test_7_insert_greater_than_no_split
[       OK ] test_7_insert_greater_than_no_split
[ RUN      ] test_8_insert_duplicate
[       OK ] test_8_insert_duplicate
[ RUN      ] test_9_insert_fill_and_split
[       OK ] test_9_insert_fill_and_split
[ RUN      ] test_12_insert_fill_split_and_grow
[       OK ] test_12_insert_fill_split_and_grow
[ RUN      ] test_13_insert_fill_split_and_grow_inverse
[       OK ] test_13_insert_fill_split_and_grow_inverse
[ RUN      ] test_14_insert_random
[       OK ] test_14_insert_random
[ RUN      ] test_15_search_none
[       OK ] test_15_search_none
[ RUN      ] test_16_insert_and_retrieve
[       OK ] test_16_insert_and_retrieve
[ RUN      ] test_18_delete_single_value
[       OK ] test_18_delete_single_value
[ RUN      ] test_19_delete_non_existant
[       OK ] test_19_delete_non_existant
[ RUN      ] test_20_delete_non_branch_key
[       OK ] test_20_delete_non_branch_key
[ RUN      ] test_21_delete_redist_left_leaf
[       OK ] test_21_delete_redist_left_leaf
[ RUN      ] test_22_delete_redist_right_leaf
[       OK ] test_22_delete_redist_right_leaf
[ RUN      ] test_22_5_redist_left_borrow
[       OK ] test_22_5_redist_left_borrow
[ RUN      ] test_23_delete_right_merge
[       OK ] test_23_delete_right_merge
[ RUN      ] test_24_delete_left_merge
[       OK ] test_24_delete_left_merge
[ RUN      ] test_25_delete_all_compress_root
[       OK ] test_25_delete_all_compress_root
[ RUN      ] test_26_delete_right_branch_merge
[       OK ] test_26_delete_right_branch_merge
[ RUN      ] test_27_delete_left_branch_merge
[       OK ] test_27_delete_left_branch_merge
[ RUN      ] test_28_insert_and_delete_random
[       OK ] test_28_insert_and_delete_random
[ RUN      ] test_29_insert_and_delete_random_large
[       OK ] test_29_insert_and_delete_random_large
[ RUN      ] test_30_insert_and_delete_strings
[       OK ] test_30_insert_and_delete_strings
[==========] 26 test(s) run.
[  PASSED  ] 26 test(s).
[==========] Running 16 test(s).
[ RUN      ] test_31_map
[       OK ] test_31_map
[ RUN      ] test_32_build_simple_tree
[       OK ] test_32_build_simple_tree
[ RUN      ] test_33_build_small_tree
[       OK ] test_33_build_small_tree
[ RUN      ] test_34_build_large_tree
[       OK ] test_34_build_large_tree
[ RUN      ] test_35_set_difference_1
[       OK ] test_35_set_difference_1
[ RUN      ] test_35_set_difference_2
[       OK ] test_35_set_difference_2
[ RUN      ] test_35_set_difference_3
[       OK ] test_35_set_difference_3
[ RUN      ] test_36_set_union_1
[       OK ] test_36_set_union_1
[ RUN      ] test_36_set_union_2
[       OK ] test_36_set_union_2
[ RUN      ] test_36_set_union_3
[       OK ] test_36_set_union_3
[ RUN      ] test_37_set_intersect_1
[       OK ] test_37_set_intersect_1
[ RUN      ] test_37_set_intersect_2
[       OK ] test_37_set_intersect_2
[ RUN      ] test_37_set_intersect_3
[       OK ] test_37_set_intersect_3
[ RUN      ] test_38_set_compliment_1
[       OK ] test_38_set_compliment_1
[ RUN      ] test_38_set_compliment_2
[       OK ] test_38_set_compliment_2
[ RUN      ] test_39_set_filter
[       OK ] test_39_set_filter
[==========] 16 test(s) run.
[  PASSED  ] 16 test(s).
[==========] Running 21 test(s).
[ RUN      ] test_1_cow_init
[       OK ] test_1_cow_init
[ RUN      ] test_2_begin_rotxn
[       OK ] test_2_begin_rotxn
[ RUN      ] test_3_begin_wrtxn_no_read
[       OK ] test_3_begin_wrtxn_no_read
[ RUN      ] test_4_begin_wrtxn_w_read
[       OK ] test_4_begin_wrtxn_w_read
[ RUN      ] test_misuse_rotxn
[       OK ] test_misuse_rotxn
[ RUN      ] test_basic_insert
[       OK ] test_basic_insert
[ RUN      ] test_large_insert
[       OK ] test_large_insert
[ RUN      ] test_random_insert
[       OK ] test_random_insert
[ RUN      ] test_out_of_order_txn_close
[       OK ] test_out_of_order_txn_close
[ RUN      ] test_dangling_txn_close
[       OK ] test_dangling_txn_close
[ RUN      ] test_txn_abort
[       OK ] test_txn_abort
[ RUN      ] test_txn_atomic_retrieve
[       OK ] test_txn_atomic_retrieve
[ RUN      ] test_txn_double_close
[       OK ] test_txn_double_close
[ RUN      ] test_txn_post_commit_use
[       OK ] test_txn_post_commit_use
[ RUN      ] test_null_txn
[       OK ] test_null_txn
[ RUN      ] test_txn_delete_simple
[       OK ] test_txn_delete_simple
[ RUN      ] test_txn_delete_leaf_left
[       OK ] test_txn_delete_leaf_left
[ RUN      ] test_txn_delete_leaf_right
[       OK ] test_txn_delete_leaf_right
[ RUN      ] test_txn_delete_branch_left
[       OK ] test_txn_delete_branch_left
[ RUN      ] test_txn_delete_branch_right
[       OK ] test_txn_delete_branch_right
[ RUN      ] test_cow_update
[       OK ] test_cow_update
[==========] 21 test(s) run.
[  PASSED  ] 21 test(s).
[==========] Running 8 test(s).
[ RUN      ] test_1_queue_invalid_create
[       OK ] test_1_queue_invalid_create
[ RUN      ] test_2_queue_enqueue
[       OK ] test_2_queue_enqueue
[ RUN      ] test_3_queue_enqueue_multiple
[       OK ] test_3_queue_enqueue_multiple
[ RUN      ] test_4_queue_invalid_dequeue
[       OK ] test_4_queue_invalid_dequeue
[ RUN      ] test_5_queue_dequeue
[       OK ] test_5_queue_dequeue
[ RUN      ] test_6_queue_dequeue_multiple
[       OK ] test_6_queue_dequeue_multiple
[ RUN      ] test_7_queue_random
[       OK ] test_7_queue_random
[ RUN      ] test_8_queue_implicit_free
[       OK ] test_8_queue_implicit_free
[==========] 8 test(s) run.
[  PASSED  ] 8 test(s).
[==========] Running 9 test(s).
[ RUN      ] test_1_tqueue_invalid_create
[       OK ] test_1_tqueue_invalid_create
[ RUN      ] test_2_tqueue_enqueue
[       OK ] test_2_tqueue_enqueue
[ RUN      ] test_3_tqueue_enqueue_multiple
[       OK ] test_3_tqueue_enqueue_multiple
[ RUN      ] test_4_tqueue_invalid_dequeue
[       OK ] test_4_tqueue_invalid_dequeue
[ RUN      ] test_5_tqueue_dequeue
[       OK ] test_5_tqueue_dequeue
[ RUN      ] test_6_tqueue_dequeue_multiple
[       OK ] test_6_tqueue_dequeue_multiple
[ RUN      ] test_7_tqueue_random
[       OK ] test_7_tqueue_random
[ RUN      ] test_8_tqueue_implicit_free
[       OK ] test_8_tqueue_implicit_free
[ RUN      ] test_9_tqueue_parallel_stress
[       OK ] test_9_tqueue_parallel_stress
[==========] 9 test(s) run.
[  PASSED  ] 9 test(s).
[==========] Running 9 test(s).
[ RUN      ] test_1_lqueue_invalid_create
[       OK ] test_1_lqueue_invalid_create
[ RUN      ] test_2_lqueue_enqueue
[       OK ] test_2_lqueue_enqueue
[ RUN      ] test_3_lqueue_enqueue_multiple
[       OK ] test_3_lqueue_enqueue_multiple
[ RUN      ] test_4_lqueue_invalid_dequeue
[       OK ] test_4_lqueue_invalid_dequeue
[ RUN      ] test_5_lqueue_dequeue
[       OK ] test_5_lqueue_dequeue
[ RUN      ] test_6_lqueue_dequeue_multiple
[       OK ] test_6_lqueue_dequeue_multiple
[ RUN      ] test_7_lqueue_random
[       OK ] test_7_lqueue_random
[ RUN      ] test_8_lqueue_implicit_free
[       OK ] test_8_lqueue_implicit_free
[ RUN      ] test_9_lqueue_parallel_stress
[       OK ] test_9_lqueue_parallel_stress
[==========] 9 test(s) run.
[  PASSED  ] 9 test(s).
[==========] Running 1 test(s).
[ RUN      ] test_siphash
hashout == test_b
src/libsds/test/test_sds_csiphash.c:44: error: Failure!
[  FAILED  ] test_siphash
[==========] 1 test(s) run.
[  PASSED  ] 0 test(s).
[  FAILED  ] 1 test(s), listed below:
[  FAILED  ] test_siphash
 1 FAILED TEST(S)
[==========] Running 8 test(s).
[ RUN      ] test_ht_null_init
[       OK ] test_ht_null_init
[ RUN      ] test_ht_null_insert
[       OK ] test_ht_null_insert
[ RUN      ] test_ht_simple_insert
[       OK ] test_ht_simple_insert
[ RUN      ] test_ht_simple_search
[       OK ] test_ht_simple_search
[ RUN      ] test_ht_medium_insert
[       OK ] test_ht_medium_insert
[ RUN      ] test_ht_large_insert
[       OK ] test_ht_large_insert
[ RUN      ] test_ht_small_delete
[       OK ] test_ht_small_delete
[ RUN      ] test_ht_medium_delete
[       OK ] test_ht_medium_delete
[==========] 8 test(s) run.
[  PASSED  ] 8 test(s).


I will reserve ppc64 machine for the further investigation.

Comment 15 wibrown@redhat.com 2017-10-24 23:24:50 UTC
Viktor, I think this issue is similar to one raised to me by someone on pagure as the siphash code makes a pointer width assumption that is not valid. There is a mailing list thread about this on 389-devel. I'm opening a ticket about it today,

Comment 16 wibrown@redhat.com 2017-10-25 00:13:11 UTC
https://pagure.io/389-ds-base/issue/49424

Comment 17 mreynolds 2017-11-03 13:59:04 UTC
Created attachment 1347404 [details]
log for build failures with cmokca tests

Here are the logs for ppc and ppc64 build failures

Comment 18 mreynolds 2017-11-03 14:00:09 UTC
The nunc-stans stress test is now randomly failing on ppc and ppc64, so I had to disable the cmocka tests for now.  See attached build log.

Comment 24 Viktor Ashirov 2018-01-15 11:25:17 UTC
According to build logs, tests are now executed during build on 64-bit platforms.

Marking as VERIFIED.

Comment 28 errata-xmlrpc 2018-04-10 14:16:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0811