Bug 1607290

Summary: qsf fails to build with gdbm support
Product: [Fedora] Fedora Reporter: Miroslav Lichvar <mlichvar>
Component: gdbmAssignee: Marek Skalický <mskalick>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: hhorak, kasal, mlichvar, mskalick, pkubat
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: gdbm-1.17-2.fc29 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-08-21 10:53:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 1606046    

Description Miroslav Lichvar 2018-07-23 08:24:39 UTC
Description of problem:
The qsf package (a spam filter) has a test suite that runs in %check. It seems to randomly fail in some GDBM tests. The qsf code didn't change in a very long time.

Some failures:
https://kojipkgs.fedoraproject.org//work/tasks/5957/28535957/build.log
https://kojipkgs.fedoraproject.org//work/tasks/5957/28535957/build.log
https://kojipkgs.fedoraproject.org//work/tasks/6077/28536077/build.log

Version-Release number of selected component (if applicable):
qsf-1.2.11-7.fc29
gdbm-1:1.16-2.fc29

Comment 1 Petr Kubat 2018-07-24 07:52:49 UTC
There was a recent rebase of gdbm in rawhide and I know that it introduced a failure in the perl testsuite at least:

https://rt.perl.org/Public/Bug/Display.html?id=133295#txn-1563420
https://puszcza.gnu.org.ua/bugs/?399

Might be something similar in qsf.

Comment 2 Jan Kurik 2018-08-14 08:42:23 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 29 development cycle.
Changing version to '29'.

Comment 3 Marek Skalický 2018-08-15 13:28:18 UTC
Problematic part of test suite is 'test/t04-training' and 'test/t05-trainvalid'. With gdbm 1.14 it's working and with gdbm-1.15 tests are failing.

Gdbm 1.15 changes can be found in http://git.gnu.org.ua/cgit/gdbm.git/tree/NEWS

It seems that training of the database is problematic. With newer gdbm, qsf keeps reclassifying spam mbox (see output below). With older gdbm, there is no reclassification in round 3 and later.

# ./qsf-1.15 -d GDBM:/tmp/qsftest2.db -T .test-spam .test-non-spam
Counting messages in folders... 15 15
round 1: checking spam...     reclassified [6.67%] 1/15
round 1: checking non-spam... reclassified [13.33%] 2/15
round 2: checking spam...     reclassified [93.33%] 14/15
round 2: checking non-spam... reclassified [0.00%] 0/15
round 3: checking spam...     reclassified [100.00%] 15/15
round 3: checking non-spam... reclassified [0.00%] 0/15
round 4: checking spam...     reclassified [100.00%] 15/15
round 4: checking non-spam... reclassified [0.00%] 0/15
round 5: checking spam...     reclassified [100.00%] 15/15
round 5: checking non-spam... reclassified [0.00%] 0/15
round 6: checking spam...     reclassified [100.00%] 15/15
round 6: checking non-spam... reclassified [0.00%] 0/15
round 7: checking spam...     reclassified [100.00%] 15/15
round 7: checking non-spam... reclassified [0.00%] 0/15
round 8: checking spam...     reclassified [100.00%] 15/15
round 8: checking non-spam... reclassified [0.00%] 0/15
round 9: checking spam...     reclassified [100.00%] 15/15
round 9: checking non-spam... reclassified [0.00%] 0/15
round 10: checking spam...     reclassified [100.00%] 15/15
round 10: checking non-spam... reclassified [0.00%] 0/15
round 11: checking spam...     reclassified [100.00%] 15/15
round 11: checking non-spam... reclassified [0.00%] 0/15
round 12: checking spam...     reclassified [100.00%] 15/15
round 12: checking non-spam... reclassified [0.00%] 0/15
round 13: checking spam...     reclassified [100.00%] 15/15
round 13: checking non-spam... reclassified [0.00%] 0/15
Several rounds with no improvement, ending training.
Optimising database... done


Unfortunately, I don't know qsf. So it's hard for me to figure out what is going on during training and why there is 'reclassified' in output. So I can't even guess what change in gdbm could cause this behavior. (I've checked qsf code related to gdbm and I don't see anything totally wrong)

Please take a look in this. I'm ready to help. Feel free to ask.

Comment 4 Miroslav Lichvar 2018-08-15 15:44:03 UTC
For me, it fails earlier in t03-marknonspam. When the db contains bad data, the following tests may fail too.

After some difficulties in bisecting due to unrelated bugs, it seems the commit that caused qsf to fail is:

http://git.savannah.gnu.org/cgit/gdbm.git/commit/?id=655cd193549e20ea8a8e77125adec7c5909c067e

More specifically, it's the first hunk in the _gdbm_get_bucket() function. If I change the code to ignore the return value of gdbm_dir_entry_valid_p(), the qsf test suite works again.

My guess is that either this check is incorrect, or qsf is doing something that corrupts the file. 

Any hints?

Comment 5 Miroslav Lichvar 2018-08-15 15:55:20 UTC
More specifically, in the case when the check fails dbf->dir[dir_index] is 512, which is less than two blocks of 512 bytes. So, the minimal change to make qsf working again is to remove the "2*" from the gdbm_dir_entry_valid_p() function.

Marek, could you please ask the upstream if that makes any sense to them?

Comment 6 Marek Skalický 2018-08-16 06:35:31 UTC
(In reply to Miroslav Lichvar from comment #4)
> For me, it fails earlier in t03-marknonspam. When the db contains bad data,
> the following tests may fail too.

But t04-training has 'rm -f $TESTDB trace'. So how t03 influences t05 if data are removed in t04?

> 
> After some difficulties in bisecting due to unrelated bugs, it seems the
> commit that caused qsf to fail is:
> 
> http://git.savannah.gnu.org/cgit/gdbm.git/commit/
> ?id=655cd193549e20ea8a8e77125adec7c5909c067e
> 
> More specifically, it's the first hunk in the _gdbm_get_bucket() function.
> If I change the code to ignore the return value of gdbm_dir_entry_valid_p(),
> the qsf test suite works again.
>
> More specifically, in the case when the check fails dbf->dir[dir_index] is
> 512, which is less than two blocks of 512 bytes. So, the minimal change to
> make qsf working again is to remove the "2*" from the
> gdbm_dir_entry_valid_p() function.
> 
> Marek, could you please ask the upstream if that makes any sense to them?

Upstream bug report - https://puszcza.gnu.org.ua/bugs/index.php?402

Comment 7 Marek Skalický 2018-08-21 10:48:16 UTC
I've build gdbm for f29 and Rawhide with backported fix introduced by upstream in mentioned bug report.

Also I've resubmitted failed qsf rawhide build to use new gdbm. qsf is building fine again in Rawhide now.

Comment 8 Miroslav Lichvar 2018-08-21 10:53:12 UTC
Great. Thanks for the update.