Bug 979784 - mongodb 2.4 available upstream
mongodb 2.4 available upstream
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: mongodb (Show other bugs)
18
Unspecified Linux
unspecified Severity unspecified
: ---
: ---
Assigned To: Nathaniel McCallum
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-30 09:57 EDT by Johan Hedin
Modified: 2013-09-23 20:32 EDT (History)
5 users (show)

See Also:
Fixed In Version: mongodb-2.4.6-1.fc19
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-23 20:32:06 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
spec-file for mongodb 2.4.4 (17.94 KB, text/plain)
2013-06-30 09:57 EDT, Johan Hedin
no flags Details
boost version fixes (11.88 KB, patch)
2013-06-30 10:00 EDT, Johan Hedin
no flags Details | Diff
mongodb-2.4.4-boost-size-fix.patch (2.16 KB, patch)
2013-06-30 10:02 EDT, Johan Hedin
no flags Details | Diff
mongodb-2.4.4-full-flag.patch (560 bytes, patch)
2013-06-30 10:03 EDT, Johan Hedin
no flags Details | Diff
mongodb-2.4.4-no-term.patch (451 bytes, patch)
2013-06-30 10:04 EDT, Johan Hedin
no flags Details | Diff
mongodb-2.4.4-shared-library-fix.patch (1.31 KB, patch)
2013-06-30 10:05 EDT, Johan Hedin
no flags Details | Diff
mongodb-2.4.4-use-system-version.patch (1.62 KB, patch)
2013-06-30 10:06 EDT, Johan Hedin
no flags Details | Diff
mongodb-2.4.4-boost-fix.patch (11.88 KB, patch)
2013-06-30 10:09 EDT, Johan Hedin
no flags Details | Diff
mongodb.init-fix1.patch (2.29 KB, patch)
2013-07-04 15:40 EDT, Johan Hedin
no flags Details | Diff
mongodb.spec-fix1.patch (6.37 KB, patch)
2013-07-04 15:41 EDT, Johan Hedin
no flags Details | Diff
mongodb-2.4.4-gcc48.patch (2.75 KB, patch)
2013-07-07 13:38 EDT, Johan Hedin
no flags Details | Diff
mongodb.spec-fix2.patch (1.20 KB, patch)
2013-07-07 13:39 EDT, Johan Hedin
no flags Details | Diff

  None (edit)
Description Johan Hedin 2013-06-30 09:57:10 EDT
Created attachment 767092 [details]
spec-file for mongodb 2.4.4

mongodb 2.4 is available upstream. 2.4 require some changes to the current spec-file to build properly and also a set of new patches.

I'm attaching a spec-file together with a set of patches that are required to build mongodb 2.4.4 on Fedora. This setup builds cleanly on Fedora 17 i363/x86_64, Fedora 18 x86_64 and CentOS 6.4 x86_64. Have not tested on Fedora 19.

The patches from 2.2.4 that are still used in this setup have been re-based to 2.4.4

I have not ported the patch for ARM since it didn't apply cleanly and I have no ARM machine to test on. Hopefully the ARM-people will be able to port that patch to 2.4.4!

Regards Johan
Comment 1 Johan Hedin 2013-06-30 10:00:15 EDT
Created attachment 767094 [details]
boost version fixes

Patch to make mongodb 2.4.4 build on multiple versions of boost.
Comment 2 Johan Hedin 2013-06-30 10:02:18 EDT
Created attachment 767095 [details]
mongodb-2.4.4-boost-size-fix.patch

Patch for boost::size()
Comment 3 Johan Hedin 2013-06-30 10:03:43 EDT
Created attachment 767096 [details]
mongodb-2.4.4-full-flag.patch

Ported patch from 2.2.4
Comment 4 Johan Hedin 2013-06-30 10:04:32 EDT
Created attachment 767097 [details]
mongodb-2.4.4-no-term.patch

Ported patch from 2.2.4
Comment 5 Johan Hedin 2013-06-30 10:05:45 EDT
Created attachment 767098 [details]
mongodb-2.4.4-shared-library-fix.patch

Patch to build libmongodb.so properly.
Comment 6 Johan Hedin 2013-06-30 10:06:36 EDT
Created attachment 767099 [details]
mongodb-2.4.4-use-system-version.patch

Ported patch from 2.2.4
Comment 7 Johan Hedin 2013-06-30 10:09:48 EDT
Created attachment 767100 [details]
mongodb-2.4.4-boost-fix.patch

Just a rename of the boost version fix patch.
Comment 8 Johan Hedin 2013-06-30 10:33:26 EDT
Hmmm, apparently the file name of the attachments are not preserved when you download them, but I guess you understand what names they should have based on the comments :-)
Comment 9 Troy Dawson 2013-07-01 15:59:58 EDT
Thank you very much Johan
I was able to download and build this for EL6, F17 and F18.
The build failed on F19 and F20, both with the same errors.

EL6
http://koji.fedoraproject.org/koji/taskinfo?taskID=5563265
F17
http://koji.fedoraproject.org/koji/taskinfo?taskID=5563262
F18
http://koji.fedoraproject.org/koji/taskinfo?taskID=5563257
F19
http://koji.fedoraproject.org/koji/taskinfo?taskID=5563252
F20 (rawhide)
http://koji.fedoraproject.org/koji/taskinfo?taskID=5563247

F19 and F20 has boost version 1.53, versus 1.50.  But it might also be that because of dependencies changing, some packages aren't installed in F19.  I'm looking into both of those.

Also, I noticed that you didn't give yourself credit in the changelog.  Did you want the changes to be anonymous, other than this bugzilla, or do you mind if I put your information in the changelog?
Comment 10 Johan Hedin 2013-07-01 16:25:37 EDT
Troy, good to hear that everything (almost) build Ok for you as well!

You could add my name and email to the spec-file, it was just an over sight since I have moved the spec file around on a lot of machines both at home and at work and cleaned it a bit in the process.

I also hope that you will solve the remaining issues with Fedora 19/20 without too much effort. I will install a Fedora 19 machine when it's available and try it out my self. I know that others have build mongodb with boost 1.53 but I don't know it they added some patches to do so.

Now to a couple of more things to continue with:

* Whats the purpose of mongodb-2.4.4-no-term.patch? I have skipped that patch my self since the rpm builds without it for me. Do we still need it? Or does it has something to do with the Fedora build system?

* I think we should wait for 2.4.5 before releasing a new package. 2.4.5 is due July 18:th and will enable us to skip the shared client build patch and perhaps some of the other patches. I will give 2.4.5-rc1 a try and see what changes needed.

* There are a couple of bugs in the spec-file and the mongod init script that I would like to get into the package as well. Just thought that I start with getting the rpm build properly with minimal changes first. Do you want those changes tracked in this ticket as well or in a new one?

Regards Johan
Comment 11 Troy Dawson 2013-07-01 18:45:11 EDT
Johan,
I still haven't gotten the packaqe to build on F19 or F20.  I'll give it another shot tomorrow.
I'd added a patch that I worked on that allows you to pass cpp and library tags to the compiler.  I created the patch so that the hardening would work.
Here is the src.rpm with the patch and the hardening in it.
http://tdawson.fedorapeople.org/mongodb/mongodb-2.4.4-2.fc20.src.rpm
http://tdawson.fedorapeople.org/mongodb/mongodb.spec
http://tdawson.fedorapeople.org/mongodb/mongodb-2.4.4-pass-flags.patch

The no-term patch was before my time with the package.  I believe someone has told me what it's for but I haven't found that yet.  It didn't have anything to do with building the package.

I'm all for waiting for 2.4.5 to come out if it's going to come out soon.  That will give us time to work off rough edges.  Maybe we can get the arm guys to rework their patch.  You probrubly noticed, it's not that the patch doesn't apply, it's that everything breaks when it's added.
And, I'm for doing all the work in this bug until we get this version released.
Troy
Comment 12 Johan Hedin 2013-07-04 15:37:50 EDT
Troy,

A quick look at the build errors on Fedora 19 indicates that the errors are related to gcc 4.8. I'm hoping to spin up a Fedora 19 machine in the following days so that I can start looking into this this as well.

As for the other improvements I would like to push into the package, I'm attaching two patches below. The patches are for mongodb.init and mongodb.spec and are patches to the files in the src.rpm you put up on fedorapeople.com.

These two patches fixes the following:

* Improvements to the init script
In its current form, it will only give mongod three seconds to shut down before killing it. This is far to short on a loaded production machine and basically means that mongod receives a SIGKILL in the middle of a sensitive fsync potentially corrupting the data base files on disk. My patch adds a wait loop that will give mongod up to five minutes to shut down but exits directly if it shuts down sooner than that.

I'm also changing the order of start/stop so that mongod start late in the boot sequence and stops early. In the old form mongod started before ntpd making replica set sync fail in case the time on the machine have not been stabilized before mongod starts.

* Renaming mongodb-devel to libmongodb-devel
I think that it makes more sense to name the development package libmongodb-devel since it is for building applications with the library, not building the database. I have done the name change in a backwards compatible way so that users will get the new package automatically. Perhaps we also need a

Provides:       mongodb-devel = %{version}-%{release}

to the libmongodb-devel package declaration, but I'm not 100% sure. Perhaps you have some insight in this?

* A lot of small fixes in the spec-file
Dependency cleanup between the packages and rpmlint warning fixes. Please see the list of fixes in the changelog.

So, does the suggested changes make sense to you? I'm happy to get some feedback!

Also, I'm going to consider your files in http://tdawson.fedorapeople.org/mongodb as the "main" repository for the iterations we are now doing so that we don't get out of sync. So if everything looks good, please update you files with the patches and I will continue from there!

A coarse list of the other improvements that I like to push into the package are:
* Bugfixes in the logging/logrotate handling
* Making a proper package for mongos
* More consistent naming of things when a mongos sub package is introduces

I'm waiting for you to comment on this first “batch” though so that we take manageable steps.

Johan
Comment 13 Johan Hedin 2013-07-04 15:40:23 EDT
Created attachment 768975 [details]
mongodb.init-fix1.patch

Fixes to the init start/stop script to make it behave correctly when shutting down a loaded database.
Comment 14 Johan Hedin 2013-07-04 15:41:39 EDT
Created attachment 768976 [details]
mongodb.spec-fix1.patch

Cleanup and fixes to the spec-file. Renamed mongodb-devel to libmongodb-devel.
Comment 15 Johan Hedin 2013-07-07 13:35:54 EDT
So, the compile problem on Fedora 19 is caused by how GCC 4.8 treat unused typedefs in some situations.

I have created a patch that fixes this in the same way as suggested on a couple of places on the net, including the gcc bugtrack. I'm attaching the patch and a patch to the spec file to incorporate the gcc patch.

I have successfully compiled the package with this patch on Fedora 17 i386/x86_64, Fedora 19 x86_64 and CentOS 6.4 x86_64 so it looks like the patch does not interfere with older compilers.

Regards Johan
Comment 16 Johan Hedin 2013-07-07 13:38:06 EDT
Created attachment 770102 [details]
mongodb-2.4.4-gcc48.patch

Patch to make mongodb compile with GCC 4.8
Comment 17 Johan Hedin 2013-07-07 13:39:09 EDT
Created attachment 770103 [details]
mongodb.spec-fix2.patch

Patch the the spec file to add the GCC 4.8 patch.
Comment 18 Troy Dawson 2013-07-08 16:51:54 EDT
I have gotten your patches and changes put together.  They are here

http://tdawson.fedorapeople.org/mongodb/mongodb-2.4.4-4.fc20.src.rpm
http://tdawson.fedorapeople.org/mongodb/mongodb.2.4.4-4.spec

Thank you so much for the spec file changes.  Looking at some of them I'm surprised that nobody complained before.
The only change I think I would make would be to make

Obsoletes:      mongodb-devel <= 2.2.4-2
to
Obsoletes:      mongodb-devel < 2.4

That would make the break at a 2.4, which is much cleaner.

I am currently working on the update to 2.4.5.  This needs to be done due to  CVE-2013-4650
Not all of the patches are applying cleanly, but I should have it done by the end of the day.
Comment 19 Johan Hedin 2013-07-08 17:13:30 EDT
Obsoletes: mongodb-devel < 2.4 sounds perfectly ok for me. Go ahead and change that!

With 2.4.5 you should drop Patch4 and also check if you can drop Patch3 as well.

Also, you will need to create a patch so that libmongoclient.so gets installed in /usr/lib64 on x86_64 since the install script in 2.4.5 installs it in /usr/lib for all architectures. See https://jira.mongodb.org/browse/SERVER-10049.

I'm also quite sure that you can drop the code in the spec file that removes the extra install of header files since this should be fixed in 2.4.5.
Comment 20 Troy Dawson 2013-07-08 18:07:26 EDT
http://tdawson.fedorapeople.org/mongodb/mongodb-2.4.5-1.fc20.src.rpm
http://tdawson.fedorapeople.org/mongodb/mongodb.2.4.5-1.spec
http://tdawson.fedorapeople.org/mongodb/mongodb.spec

Yep, patches 3 and 4 could be dropped, which I did.
I did a 'sed' instead of a patch for the /usr/lib problem.  A patch isn't dynamic, so it can't tell where to put things during build time.
I also commented out the fix for the extra header problem we had.

I haven't had time for a full build of all of these, I've only verified that the patches and sed do what was expected of them.  But I have them building now, we'll know in a while if everything is fixed and working.
Comment 21 Johan Hedin 2013-07-09 16:50:37 EDT
Great!

The sed approach was what I was thinking about for exactly the same reason :-)

I'm in the process of breaking down the rest of my suggestions in manageable chunks and will post here as soon as possible.
Comment 22 Troy Dawson 2013-07-10 16:53:26 EDT
Two comments/concerns.

Have you tested the new init.d script on both single node mongo servers, as well as sharded servers?

What is the reasoning for wanting to break out mongos?  Can you do sharding without the server?
I'm very concerned that this will start breaking users sharded machines when they do an update, and I don't see a reason for it.
Comment 23 Troy Dawson 2013-07-12 17:36:36 EDT
I have worked with David Marlin and he got the arm patches fixed up and working.
After much trial and error I found that the hardening breaks 2.4.x.  I had the if statements messed up, so it really wasn't calling the hardening stuff.  Tracking it down, it has to do with the s2 code.
I have alot of people asking for mongodb 2.4.x in any form, and very few asking for the hardening stuff.
I am leaving in the patch that allows us to do the hardening, but not calling the hardening in the spec file.  In this format I have tested it across f18-f20, EPEL6, and on arm f18-f20.  It works for all of those.
I am going to commit this, just to rawhide.  That will allow those that really want it to use it and test it.
We need to find out how backwards compatible it is, and how easy/hard it will be to push it to F18,F19 and EPEL6.
http://tdawson.fedorapeople.org/mongodb/mongodb-2.4.5-3.fc20.src.rpm
http://tdawson.fedorapeople.org/mongodb/mongodb.2.4.5-3.spec
http://tdawson.fedorapeople.org/mongodb/mongodb.spec

Troy
Comment 24 Johan Hedin 2013-07-14 10:59:29 EDT
Troy,

Your changes and way forward sounds ok. Hardening is not as important for me (running mongodb on a separate network) so I leave that to others to argue for.

As for your questions about mongos/sharding/mongod:

I'm using mongodb in both stand alone, replica set and sharded environments with rpm:s that includes everything I'm suggesting here. So the changes are battle proven for the last three-four months, including the init.d script. Upgrade paths from a 2.2 package from EPEL/Fedora to my own packages is something that I have not addressed all the way though since I wasn't in need of it.

The reason for wanting to break out mongos and put it in its own sub-package is that mongos is a service you normally only run on your app-servers. Bringing in the whole database on those servers is not necessary. In the same way there is no need for the mongos binary to be installed on every machine that runs mongod (not even when the mongod:s are part of a sharded setup).

So it's a matter of getting just the right amount of files needed on a machine for it to do it's job.

You are right though that this i a bit backwards breaking and needs to be addressed. Someone that have installed mongodb-server on their app-server just to get the mongos binary will perhaps be surprised to see the the mongos binary disappeared when doing an update.

Do you know of any standard way/pattern to let RPM bring in a new package on an update without a need for a Requires-directive? I'm thinking if triggers could be used so that a user updating mongodb-server from 2.2 to 2.4 will get the new package for mongos installed as well, while a user installing mongodb-server 2.4 for the first time will not.

To not get stuck with this issue, I suggest that we at least move on and bring in the necessary files in the mongodb-server package for making a proper service of mongos (init.d script, logrotate script, systemd service unit etc). Then we can see if a separate package for mongos is doable or not.

Before bringing in the new files, I would like to bring up a suggestion for some name changes. This is also a bit backwards breaking, but makes things clearer in my opinion.

As it is now, the following files:

/etc/mongodb.conf
/etc/logrotate.d/mongodb
/var/run/mongodb/mongodb.pid
/var/log/mongodb/mongodb.log

all have the name mongodb.* even though they relate to the binary/service mongod. When bringing in mongos in the game it makes more sense to rename all these files to mongod.* instead.

I realize that a name change on /etc/mongodb.conf is perhaps a bit to much to ask for since it's probably confusing for users upgrading from 2.2 but what do you think about the others? With such a change we would have:

/etc/mongodb.conf
/etc/logrotate.d/mongod
/var/run/mongodb/mongod.pid
/var/log/mongodb/mongod.log

for mongod.

For mongos I'm suggest that we run the mongos service as the mongodb user and re-use the base directories from mongod for mongos. Thus, for mongos we will have the following new files added to the package:

/etc/mongos.conf
/etc/logrotate.d/mongos
/var/run/mongodb/mongos.pid
/var/log/mongodb/mongos.log
/usr/lib/systemd/system/mongos.service

The current logrotate script for mongod rotates all files in /var/log/mongodb so that has to be changed so that /etc/logrotate.d/mongod only rotates /var/log/mongodb/mongod.log and the new /etc/logrotate.d/mongos only rotates /var/log/mongodb/mongod.log.

/etc/logrotate.d/mongodb is marked as %config(noreplace) so changing that file could possibly have side effects for users that have changed the file themselves.

Regardless of the above, the current logrotate script does not clean up /var/log/mongodb properly since mongod itself rename files when rotating them in a way  that the logrotate script does not handle. So fixing this require changes to the current logrotate script anyhow.

Finally, I'm suggesting that we drop /etc/sysconfig/mongod. mongod does not have any special command line arguments. Anything you could put on the command line could be put in the config file. Having two places to put config that does the same thing is confusing. The current /etc/init.d/mongod script is even written i such way that it does not make any use of /etc/sysconfig/mongod.

A lot of changes suggested in this post, but I was not able to break them down in manageable chunks since many of them relate to each other. I hope that you are able to at least grab some of them and I'm looking forward to your comments.

I have all the files needed for making mongos a service ready to post here as soon as we agree on how to proceed.

Johan
Comment 25 Fedora Update System 2013-09-19 16:56:58 EDT
mongodb-2.4.6-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/mongodb-2.4.6-1.fc19
Comment 26 Fedora Update System 2013-09-21 04:27:56 EDT
Package mongodb-2.4.6-1.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing mongodb-2.4.6-1.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-17283/mongodb-2.4.6-1.fc19
then log in and leave karma (feedback).
Comment 27 Fedora Update System 2013-09-23 20:32:06 EDT
mongodb-2.4.6-1.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.