Created attachment 894674 [details] log of the testsuite run Description of problem: Running the testsuite of mariadb-10.0.x on ppc64 results in many failing tests. Version-Release number of selected component (if applicable): mariadb-10.0.10-1.fc21 Check of testcase failed for: rpl.rpl_extra_col_master_myisam rpl.rpl_extra_col_master_innodb rpl.rpl_deadlock_innodb Failing test(s): connect.alter connect.bin connect.dbf connect.index multi_source.skip_counter rpl.rpl_auto_increment rpl.rpl_mixed_binlog_max_cache_size rpl.rpl_slave_skip rpl.rpl_gtid_until rpl.rpl_skip_replication rpl.rpl_stm_max_relay_size rpl.rpl_row_loaddata_concurrent rpl.rpl_extra_col_slave_innodb rpl.rpl_row_binlog_max_cache_size rpl.rpl_row_tabledefs_3innodb rpl.rpl_extra_col_slave_myisam rpl.rpl_typeconv rpl.rpl_row_max_relay_size rpl.rpl_row_tabledefs_2myisam rpl.rpl_stm_loaddata_concurrent rpl.rpl_loaddata rpl.rpl_stm_binlog_max_cache_size rpl.rpl_filter_tables_not_exist rpl.rpl_filter_tables_dynamic rpl.rpl_stm_000001 rpl.rpl_gtid_basic rpl.rpl_gtid_master_promote rpl.rpl_gtid_stop_start rpl.rpl_rotate_logs main.statistics main.gis-precise sys_vars.max_relay_log_size_basic sys_vars.sql_slave_skip_counter_basic
I think the same problems exist on s390(x) from build.log ... -------------------------------------------------------------------------- The servers were restarted 261 times Spent 495.459 of 1925 seconds executing testcases Check of testcase failed for: rpl.rpl_deadlock_innodb Too many failed: Failed 10/119 tests, 91.60% were successful. Failing test(s): multi_source.skip_counter rpl.rpl_auto_increment rpl.rpl_mixed_binlog_max_cache_size rpl.rpl_slave_skip rpl.rpl_gtid_until rpl.rpl_skip_replication The log files in var/log may give you some hint of what went wrong. If you want to report this error, please read first the documentation at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html 412 tests were skipped, 143 by the test itself. mysql-test-run: *** ERROR: there were failing test cases error: Bad exit status from /var/tmp/rpm-tmp.ruicdB (%check) RPM build errors:
with http://pkgs.fedoraproject.org/cgit/mariadb.git/tree/rh-skipped-tests-ppc64.list I was able to build mariadb-10.0.11 on s390(x) and ppc64
mariadb 10.0.1-5 failing on ppc64 as reported in http://ppc.koji.fedoraproject.org/koji/buildinfo?buildID=238271 I need to add new tests in skipped list files on top of release 10.0.12-2 as suggested below to make build ppc64/ppc64le to complete in scratch build http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=1905057 === diff --git a/rh-skipped-tests-ppc-s390.list b/rh-skipped-tests-ppc-s390.list index 8c15a79..4509693 100644 --- a/rh-skipped-tests-ppc-s390.list +++ b/rh-skipped-tests-ppc-s390.list @@ -3,6 +3,7 @@ connect.bin : rhbz#1096787 connect.dbf : rhbz#1096787 connect.index : rhbz#1096787 main.gis-precise : rhbz#1096787 +main.openssl_1 : rhbz#1096787 main.statistics : rhbz#1096787 multi_source.skip_counter : rhbz#1096787 rpl.rpl_auto_increment : rhbz#1096787 diff --git a/rh-skipped-tests-ppc64le.list b/rh-skipped-tests-ppc64le.list index 716f7b3..1fc4a2b 100644 --- a/rh-skipped-tests-ppc64le.list +++ b/rh-skipped-tests-ppc64le.list @@ -1,3 +1,5 @@ main.gis-precise : rhbz#1096787 main.mysqlslap : rhbz#1096787 +main.openssl_1 : rhbz#1096787 rpl.rpl_insert : rhbz#1096787 +vcol.vcol_supported_sql_funcs_myisam : rhbz#1096787 ===
Could ignore the previous patches suggested in comment #4 with new version mariadb-10.0.12-4.fc21 where the ppc64le archi build is failing for 3 tests as reported in Task Id http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=1960557 === extract: ... Spent 2658.015 of 8880 seconds executing testcases Completed: Failed 3/2890 tests, 99.90% were successful. Failing test(s): rpl.rpl_insert_delayed main.openssl_1 vcol.vcol_supported_sql_funcs_myisam === Able to bypass related tests on ppc64le with following git patch as reported by scratch build Id http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=1960681 === $git diff diff --git a/rh-skipped-tests-ppc64le.list b/rh-skipped-tests-ppc64le.list index 716f7b3..9cf211b 100644 --- a/rh-skipped-tests-ppc64le.list +++ b/rh-skipped-tests-ppc64le.list @@ -1,3 +1,6 @@ main.gis-precise : rhbz#1096787 main.mysqlslap : rhbz#1096787 +main.openssl_1 : rhbz#1096787 rpl.rpl_insert : rhbz#1096787 +rpl.rpl_insert_delayed : rhbz#1096787 +vcol.vcol_supported_sql_funcs_myisam : rhbz#1096787 ===
Oups comment #5 was sent too early :( sorry the Task Id 1960681 is still running.
With new version 10.0.13 two tests newly fail on armv7hl: innodb.innodb_simulate_comp_failures_small main.key_cache Disabling for now, since there is not enough time to debug.
(In reply to Michel Normand from comment #5) > +rpl.rpl_insert_delayed : rhbz#1096787 The failure of rpl_insert_delayed is caused by mysqlslap deadlocking. The issue is fixed in upstream mariadb 10.0.13 - https://mariadb.atlassian.net/browse/MDEV-6489. I can confirm it passes with mariadb-10.0.13-4.fc22. We may need to backport the patch for F21 - attaching. openssl_1 and vcol_supported_sql_funcs_myisam are still failing.
Created attachment 930782 [details] mysqlslap-fix-deadlock.patch
I think the openssl_1 started failing in mariadb-10.0.12-3.fc21, mariadb-10.0.12-2.fc21 was OK (in both ppc and s390 koji), so either the test used to be disabled, or some other change made it to fail
(In reply to Dan Horák from comment #10) > I think the openssl_1 started failing in mariadb-10.0.12-3.fc21, > mariadb-10.0.12-2.fc21 was OK (in both ppc and s390 koji), so either the > test used to be disabled, or some other change made it to fail This test might fail because of changes in openssl: * Mon Jun 30 2014 Tomáš Mráz <tmraz> 1.0.1h-5 - disable SSLv2 and SSLv3 protocols by default (can be enabled via appropriate SSL_CTX_clear_options() call) http://koji.fedoraproject.org/koji/buildinfo?buildID=540893 (In reply to Michal Toman from comment #9) > Created attachment 930782 [details] > mysqlslap-fix-deadlock.patch Great, thanks! I'm planning to sync rawhide and F21 after alpha is out, so it should get to F21 in few weeks. Also, I've just find out that some fixes for main.gis-precise test are already part of 10.0.13 -- only test is fixed, no changes necessary in the code; plus some other tests have been fixed to work properly on power8, which could cover all ppc arches. We may re-run the suite on all arches without disabling the tests (no --skip-test-list=) and see which are already fixed (not sure if I manage to do it today and I'll be out for the rest of the week, so if nobody else does it, I'll try that the next week hopefully).
(In reply to Honza Horak from comment #11) > (In reply to Dan Horák from comment #10) > > I think the openssl_1 started failing in mariadb-10.0.12-3.fc21, > > mariadb-10.0.12-2.fc21 was OK (in both ppc and s390 koji), so either the > > test used to be disabled, or some other change made it to fail > > This test might fail because of changes in openssl: > * Mon Jun 30 2014 Tomáš Mráz <tmraz> 1.0.1h-5 - disable SSLv2 and > SSLv3 protocols by default (can be enabled via appropriate > SSL_CTX_clear_options() call) > http://koji.fedoraproject.org/koji/buildinfo?buildID=540893 This is indeed true, the test passes with openssl-1.0.1h-2.fc21 and openssl-1.0.1h-3.fc21 (the -3 is not build for power) and does not with openssl-1.0.1h-4.fc21. Tested on ppc and s390.
So I found the openssl problem. Long story short, openssl unintentionally disabled the RC4 cipher which is used in mariadb's testsuite - #1134379. crypto-policies-20140827-1.git4e06f1d.fc21* should fix the problem. * http://koji.fedoraproject.org/koji/taskinfo?taskID=7466056
This bug appears to have been reported against 'rawhide' during the Fedora 22 development cycle. Changing version to '22'. More information and reason for this action is here: https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora22
New bug reports connected to failing tests: https://mariadb.atlassian.net/browse/MDEV-7973 http://bugs.mysql.com/bug.php?id=76654
This bug appears to have been reported against 'rawhide' during the Fedora 23 development cycle. Changing version to '23'. (As we did not run this process for some time, it could affect also pre-Fedora 23 development cycle bugs. We are very sorry. It will help us with cleanup during Fedora 23 End Of Life. Thank you.) More information and reason for this action is here: https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora23
This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component.
Since 10.0.24, mariadb started to fail also in main.ssl_cert_verify. So adding to the devil's list. main.ssl_cert_verify [ fail ] Test ended at 2016-06-24 15:28:03 CURRENT_TEST: main.ssl_cert_verify mysqltest: In included file "/builddir/build/BUILD/mariadb-10.0.25/mysql-test/include/wait_until_connected_again.inc": included from /builddir/build/BUILD/mariadb-10.0.25/mysql-test/t/ssl_cert_verify.test at line 20: At line 26: query 'show status' failed with wrong errno 2026: 'SSL connection error: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed', instead of 0... The result from queries just before the failure was: #T1: Host name (/CN=localhost/) as OU name in the server certificate, server certificate verification should fail. - saving '/builddir/build/BUILD/mariadb-10.0.25/mysql-test/var/log/main.ssl_cert_verify/' to '/builddir/build/BUILD/mariadb-10.0.25/mysql-test/var/log/main.ssl_cert_verify/'
Got some more tests failing with mariadb-10.1.16-1 on ppc64 and ppc64le. https://ppc.koji.fedoraproject.org/koji/packageinfo?packageID=15127 here after is (a non exhaustive I guess) list of fail tests: encryption.innodb_onlinealter_encryption encryption.innodb_encryption_filekeys encryption.innodb_encryption_discard_import innodb_fts.innodb_fts_transaction rpl.rpl_innodb_bug30888 rpl.rpl_mdev6020 innodb.innodb_mutexes rpl.rpl_gtid_master_promote encryption.innodb_onlinealter_encryption test fails everytime but other tests passed ok sometimes. main.failed_auth_unixsocket fails when user name is "test" Not sure if the problem is a thread problem or a timing problem but I didn't have fail tests when: Tests are running on a VM with qemu parameters: "-smp cpus=1,threads=1" Most of the time tests are successfull when I use command: "numactl --physcpubind=+1" ex: numactl --physcpubind=+1 ./mysql-test-run.pl... but sometime we can have one test fail randomly from the above list.
This message is a reminder that Fedora 23 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 23. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '23'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 23 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Issues with tests: * sys_vars.wsrep_provider_options_basic * main.ssl_8k_key solved in Rawhide. Recent list of remaining skipped tests (rh-skipped-tests-base.list) in Rawhide: * main.userstat : rhbz#1096787 * main.multi_update : rhbz#1096787 * main.set_statement_notembedded_binlog : rhbz#1096787 * main.ssl_7937 : rhbz#1096787 * main.ssl_crl_clients : rhbz#1096787 * main.ssl_cert_verify : rhbz#1096787 * perfschema.nesting : rhbz#1096787 * perfschema.socket_summary_by_event_name_func : rhbz#1096787 * perfschema.socket_summary_by_instance_func : rhbz#1096787
Directing back to F23, because it is the last Fedora version, that use mariadb-10.0.x For F>=24, which use mariadb 10.1.x, was created appropriate tracker └── https://bugzilla.redhat.com/show_bug.cgi?id=1399847 ______________________________________________________________________________________ In version F23 - mariadb-10.0.28 - only following tests are currently failing ├── main.ssl_7937 ├── main.ssl_crl_clients ├── main.ssl_cert_verify ├── perfschema.nesting ├── perfschema.socket_summary_by_event_name_func └── perfschema.socket_summary_by_instance_func ______________________________________________________________________________________ Changes were pushed, but waiting to be built (aggregating patches for a last build before F23 EOL )
Fedora 23 changed to end-of-life (EOL) status on 2016-12-20. Fedora 23 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.