Bug 544275 - Update bsddb3 module for db 4.8
Update bsddb3 module for db 4.8
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: python (Show other bugs)
rawhide
All Linux
low Severity medium
: ---
: ---
Assigned To: Dave Malcolm
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-12-04 08:10 EST by Josh Boyer
Modified: 2009-12-18 11:34 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-12-18 10:16:26 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Patch to update bsddb3 to 4.8.1 (112.42 KB, patch)
2009-12-04 08:10 EST, Josh Boyer
no flags Details | Diff
spec file patch (1.37 KB, patch)
2009-12-04 08:11 EST, Josh Boyer
no flags Details | Diff

  None (edit)
Description Josh Boyer 2009-12-04 08:10:41 EST
Created attachment 376071 [details]
Patch to update bsddb3 to 4.8.1

Description of problem:

The current version of python in rawhide uses bsddb3 4.7.x.  However, the Oracle DB version in rawhide has now been updated to 4.8, and python will fail to build if you try and build it against this.  You can see that in this scratch build:


Version-Release number of selected component (if applicable):

python-2.6.4-3.fc13

How reproducible:

Always

Steps to Reproduce:
1.  Build python in rawhide.
2.  Watch it fail to build

Actual results:

Compile errors.

Expected results:

Working build

Additional info:

The attached patch updates the bsddb3 module to version 4.8.1 from:

http://pypi.python.org/pypi/bsddb3/

You can see a successful scratch build of this here:

http://koji.fedoraproject.org/koji/taskinfo?taskID=1849105
Comment 1 Josh Boyer 2009-12-04 08:11:25 EST
Created attachment 376072 [details]
spec file patch
Comment 2 Josh Boyer 2009-12-04 08:12:41 EST
(In reply to comment #0)
> Created an attachment (id=376071) [details]
> Patch to update bsddb3 to 4.8.1
> 
> Description of problem:
> 
> The current version of python in rawhide uses bsddb3 4.7.x.  However, the
> Oracle DB version in rawhide has now been updated to 4.8, and python will fail
> to build if you try and build it against this.  You can see that in this
> scratch build:

Oops, forgot to put URL to failing scratch build:

http://koji.fedoraproject.org/koji/taskinfo?taskID=1849142
Comment 3 Josh Boyer 2009-12-07 15:13:21 EST
I'd be more than happy to commit this and build it if the change seems acceptable.
Comment 4 Dave Malcolm 2009-12-08 09:15:33 EST
Sorry about belated response; I'm on the road (FUDCon) and have only very intermittent connectivity right now.

Thanks for generating the patch.

Do all the tests still pass if you run:
python /usr/lib/python2.6/test/test_bsddb3.py
(part of the python-test subpackage of our python rpms)

I'm wondering how best to approach this, especially with respect to upstream.  Am I right in thinking that you're attempting to bump the version of the dependent package in rawhide?  What other packages needed work to deal with the API changes?  Is there a good overview somewhere of these changes (do you have a feature page for this work?)

I'm a little wary of carrying a big patch in our SRPM, but I'm not yet sure how best to solve this - if our aim is to keep support for the module within the "python" rpm, I wonder if would be better to simply ship the tarball from the standalone module within the python srpm, and use that code directly.

(need to get back on the bus now)
Comment 5 Jeremy Katz 2009-12-08 09:30:28 EST
Generally for new bdb versions, we've done significantly simpler patches.  See, eg, F-10/python-2.5.2-db47.patch for the one that brought us up to working with bdb4.7 in python 2.5.2
Comment 6 James Antill 2009-12-08 10:33:24 EST
 Yeh, I mentioned to Josh that previous db bumps had been much smaller but I assumed that the patch was from upstream.
 He replied it was from upstream, and the reason it was so big was because the BDB API had changed (AIUI this only affects internal python, and not any python programs).

 David, I'm pretty sure just bundling the old version will have Spot opening a BZ to stop it :).
Comment 7 Josh Boyer 2009-12-08 11:41:04 EST
(In reply to comment #4)
> Sorry about belated response; I'm on the road (FUDCon) and have only very
> intermittent connectivity right now.
> 
> Thanks for generating the patch.
> 
> Do all the tests still pass if you run:
> python /usr/lib/python2.6/test/test_bsddb3.py
> (part of the python-test subpackage of our python rpms)

I haven't tried.  I don't have a rawhide machine available at the moment.

> I'm wondering how best to approach this, especially with respect to upstream. 
> Am I right in thinking that you're attempting to bump the version of the
> dependent package in rawhide?  What other packages needed work to deal with the
> API changes?  Is there a good overview somewhere of these changes (do you have
> a feature page for this work?)

I didn't bump anything.  The owner of the Oracle DB package bumped it in rawhide from 4.7 to 4.8.  If you try to build the exising python against the new rawhide build root, it fails.  You can see that with the scratch build URL in the previous comment.

You'll hit this failure the next time you build python in rawhide, so I'm just trying to get a jump on it.

> I'm a little wary of carrying a big patch in our SRPM, but I'm not yet sure how
> best to solve this - if our aim is to keep support for the module within the
> "python" rpm, I wonder if would be better to simply ship the tarball from the
> standalone module within the python srpm, and use that code directly.

Yeah, I understand the concerns about the patch.  It is upstream, just in patch form.  If you'd rather use a tarball, that's fine with me.  You can see the overview of the changes for db4.8 here:

http://www.jcea.es/programacion/pybsddb.htm

Of particular note is the highlights for bsddb3-4.8.0
Comment 8 Dave Malcolm 2009-12-17 22:43:54 EST
CCing jnovy (db4/compat-db maintainer) 

jnovy: how substantial are the API changes from 4.7 to 4.8?

Here's a review of what we've done historically w.r.t. building python against a changing db4.

FC-4, FC-5, FC-6: no patches; "BuildPrereq: db4-devel >= 4.3" from FC-5 onwards

F-7 (Python-2.5)'s python.spec had:
  Patch6: python-db45.patch
$ diffstat ../F-7/python-db45.patch 
 Modules/_bsddb.c |    9 ++++++++-
 setup.py         |    4 ++--
 2 files changed, 10 insertions(+), 3 deletions(-)

F-8 and F-9 (both Python-2.5.1) had:
  Patch60: python-2.5.1-db46.patch
$ diffstat ../F-8/python-2.5.1-db46.patch 
 setup.py |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

Upstream merger for 4.6.4: http://svn.python.org/view?view=rev&revision=63207

F-10 (2.5.2) had:
  Patch60: python-2.5.2-db47.patch
$ diffstat ../F-10/python-2.5.2-db47.patch 
 Modules/_bsddb.c |   27 ++++++++++++++++++++++-----
 setup.py         |    4 ++--
 2 files changed, 24 insertions(+), 7 deletions(-)

F-11 (2.6) and F-12 (2.6.2) didn't have a patch for the bsddb code (or, rather, had the line from F-10 in commented-out form)

Upstream support for 4.7 happened here:
 - merging from http://www.jcea.es/programacion/pybsddb.htm#bsddb3-4.7.3 in http://svn.python.org/view?view=rev&revision=66088
 - http://www.jcea.es/programacion/pybsddb.htm#bsddb3-4.7.2 in http://svn.python.org/view?view=rev&revision=65193
 - bsddb module updated to version 4.7.0 in http://svn.python.org/view?view=rev&revision=63537


Proposed patch is clearly much larger:
$ diffstat python-2.6-update-bsddb3-4.8.patch 
 Lib/bsddb/__init__.py                           |   14 
 Lib/bsddb/dbobj.py                              |   21 
 Lib/bsddb/dbtables.py                           |    9 
 Lib/bsddb/dbutils.py                            |    2 
 Lib/bsddb/test/test_all.py                      |   44 
 Lib/bsddb/test/test_associate.py                |   66 -
 Lib/bsddb/test/test_basics.py                   |  163 ++
 Lib/bsddb/test/test_compare.py                  |   22 
 Lib/bsddb/test/test_compat.py                   |    2 
 Lib/bsddb/test/test_dbenv.py                    |  148 ++
 Lib/bsddb/test/test_dbtables.py                 |   16 
 Lib/bsddb/test/test_distributed_transactions.py |    6 
 Lib/bsddb/test/test_early_close.py              |   13 
 Lib/bsddb/test/test_fileid.py                   |   63 +
 Lib/bsddb/test/test_lock.py                     |   15 
 Lib/bsddb/test/test_pickle.py                   |   12 
 Lib/bsddb/test/test_recno.py                    |    2 
 Lib/bsddb/test/test_replication.py              |   25 
 Lib/bsddb/test/test_sequence.py                 |    8 
 Lib/bsddb/test_support.py                       |   54 
 Modules/_bsddb.c                                | 1426 ++++++++++++++++++++----
 Modules/bsddb.h                                 |   13 
 22 files changed, 1815 insertions(+), 329 deletions(-)


Currently, python-2.6.4-3 in rawhide has an autogenerated requires on "libdb-4.7.so()" which is being satisfied by compat-db47.

Running the (unpatched) python-2.6.4-3 code through /usr/lib64/python2.6/test/test_bsddb3.py shows 1 error:
======================================================================
FAIL: test01_basic_replication (bsddb.test.test_replication.DBReplicationManager)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.6/bsddb/test/test_replication.py", line 134, in test01_basic_replication
    self.assertTrue(time.time()<timeout)
AssertionError

----------------------------------------------------------------------
Ran 340 tests in 43.275s

FAILED (failures=1)

I can see three approaches here:
  (a) apply the patch as is
  (b) ship the PyBSDDB tarball within our python SRPM, and use that directly, rather than carrying a patch between the Python-2.6.4.tar.bz2's version of the code vs this external tarball
  (c) try to come up with a smaller patch
  (d) build and link against compat-db47 instead

Given that all of the prior patches came from the same source, I'm inclined to go with (a) for now.

I needed to patch things a bit more to force it to link against 4.8; scratch built at http://koji.fedoraproject.org/koji/taskinfo?taskID=1878602

This scratch build gives full passes for the test suite on my x86_64 rawhide box:
# python /usr/lib64/python2.6/test/test_bsddb3.py
  (snip)
----------------------------------------------------------------------
Ran 444 tests in 38.690s

OK

Tagged and building into dist-fc13 as python-2.6.4-4.fc13: http://koji.fedoraproject.org/koji/taskinfo?taskID=1878788
Comment 9 Jindrich Novy 2009-12-18 11:34:27 EST
Just for the record, the db-4.8.24 release doesn't contain major compatibility-breaking changes. The benefit of upgrading is handful of fixes that 4.8.24 brings.

If you are curious what changed in 4.8.24 from 4.7 BDB series, please have a look here:

http://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/changelog_4_8.html#id1655245

All changes are listed here:

http://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/changelog_4_8.html

A good thing is that there is only one function added in case of the most widely used C API. There are only bugfixes in other APIs such as C++ and Java one.

Note You need to log in before you can comment on or make changes to this bug.