Created attachment 767092 [details] spec-file for mongodb 2.4.4 mongodb 2.4 is available upstream. 2.4 require some changes to the current spec-file to build properly and also a set of new patches. I'm attaching a spec-file together with a set of patches that are required to build mongodb 2.4.4 on Fedora. This setup builds cleanly on Fedora 17 i363/x86_64, Fedora 18 x86_64 and CentOS 6.4 x86_64. Have not tested on Fedora 19. The patches from 2.2.4 that are still used in this setup have been re-based to 2.4.4 I have not ported the patch for ARM since it didn't apply cleanly and I have no ARM machine to test on. Hopefully the ARM-people will be able to port that patch to 2.4.4! Regards Johan
Created attachment 767094 [details] boost version fixes Patch to make mongodb 2.4.4 build on multiple versions of boost.
Created attachment 767095 [details] mongodb-2.4.4-boost-size-fix.patch Patch for boost::size()
Created attachment 767096 [details] mongodb-2.4.4-full-flag.patch Ported patch from 2.2.4
Created attachment 767097 [details] mongodb-2.4.4-no-term.patch Ported patch from 2.2.4
Created attachment 767098 [details] mongodb-2.4.4-shared-library-fix.patch Patch to build libmongodb.so properly.
Created attachment 767099 [details] mongodb-2.4.4-use-system-version.patch Ported patch from 2.2.4
Created attachment 767100 [details] mongodb-2.4.4-boost-fix.patch Just a rename of the boost version fix patch.
Hmmm, apparently the file name of the attachments are not preserved when you download them, but I guess you understand what names they should have based on the comments :-)
Thank you very much Johan I was able to download and build this for EL6, F17 and F18. The build failed on F19 and F20, both with the same errors. EL6 http://koji.fedoraproject.org/koji/taskinfo?taskID=5563265 F17 http://koji.fedoraproject.org/koji/taskinfo?taskID=5563262 F18 http://koji.fedoraproject.org/koji/taskinfo?taskID=5563257 F19 http://koji.fedoraproject.org/koji/taskinfo?taskID=5563252 F20 (rawhide) http://koji.fedoraproject.org/koji/taskinfo?taskID=5563247 F19 and F20 has boost version 1.53, versus 1.50. But it might also be that because of dependencies changing, some packages aren't installed in F19. I'm looking into both of those. Also, I noticed that you didn't give yourself credit in the changelog. Did you want the changes to be anonymous, other than this bugzilla, or do you mind if I put your information in the changelog?
Troy, good to hear that everything (almost) build Ok for you as well! You could add my name and email to the spec-file, it was just an over sight since I have moved the spec file around on a lot of machines both at home and at work and cleaned it a bit in the process. I also hope that you will solve the remaining issues with Fedora 19/20 without too much effort. I will install a Fedora 19 machine when it's available and try it out my self. I know that others have build mongodb with boost 1.53 but I don't know it they added some patches to do so. Now to a couple of more things to continue with: * Whats the purpose of mongodb-2.4.4-no-term.patch? I have skipped that patch my self since the rpm builds without it for me. Do we still need it? Or does it has something to do with the Fedora build system? * I think we should wait for 2.4.5 before releasing a new package. 2.4.5 is due July 18:th and will enable us to skip the shared client build patch and perhaps some of the other patches. I will give 2.4.5-rc1 a try and see what changes needed. * There are a couple of bugs in the spec-file and the mongod init script that I would like to get into the package as well. Just thought that I start with getting the rpm build properly with minimal changes first. Do you want those changes tracked in this ticket as well or in a new one? Regards Johan
Johan, I still haven't gotten the packaqe to build on F19 or F20. I'll give it another shot tomorrow. I'd added a patch that I worked on that allows you to pass cpp and library tags to the compiler. I created the patch so that the hardening would work. Here is the src.rpm with the patch and the hardening in it. http://tdawson.fedorapeople.org/mongodb/mongodb-2.4.4-2.fc20.src.rpm http://tdawson.fedorapeople.org/mongodb/mongodb.spec http://tdawson.fedorapeople.org/mongodb/mongodb-2.4.4-pass-flags.patch The no-term patch was before my time with the package. I believe someone has told me what it's for but I haven't found that yet. It didn't have anything to do with building the package. I'm all for waiting for 2.4.5 to come out if it's going to come out soon. That will give us time to work off rough edges. Maybe we can get the arm guys to rework their patch. You probrubly noticed, it's not that the patch doesn't apply, it's that everything breaks when it's added. And, I'm for doing all the work in this bug until we get this version released. Troy
Troy, A quick look at the build errors on Fedora 19 indicates that the errors are related to gcc 4.8. I'm hoping to spin up a Fedora 19 machine in the following days so that I can start looking into this this as well. As for the other improvements I would like to push into the package, I'm attaching two patches below. The patches are for mongodb.init and mongodb.spec and are patches to the files in the src.rpm you put up on fedorapeople.com. These two patches fixes the following: * Improvements to the init script In its current form, it will only give mongod three seconds to shut down before killing it. This is far to short on a loaded production machine and basically means that mongod receives a SIGKILL in the middle of a sensitive fsync potentially corrupting the data base files on disk. My patch adds a wait loop that will give mongod up to five minutes to shut down but exits directly if it shuts down sooner than that. I'm also changing the order of start/stop so that mongod start late in the boot sequence and stops early. In the old form mongod started before ntpd making replica set sync fail in case the time on the machine have not been stabilized before mongod starts. * Renaming mongodb-devel to libmongodb-devel I think that it makes more sense to name the development package libmongodb-devel since it is for building applications with the library, not building the database. I have done the name change in a backwards compatible way so that users will get the new package automatically. Perhaps we also need a Provides: mongodb-devel = %{version}-%{release} to the libmongodb-devel package declaration, but I'm not 100% sure. Perhaps you have some insight in this? * A lot of small fixes in the spec-file Dependency cleanup between the packages and rpmlint warning fixes. Please see the list of fixes in the changelog. So, does the suggested changes make sense to you? I'm happy to get some feedback! Also, I'm going to consider your files in http://tdawson.fedorapeople.org/mongodb as the "main" repository for the iterations we are now doing so that we don't get out of sync. So if everything looks good, please update you files with the patches and I will continue from there! A coarse list of the other improvements that I like to push into the package are: * Bugfixes in the logging/logrotate handling * Making a proper package for mongos * More consistent naming of things when a mongos sub package is introduces I'm waiting for you to comment on this first “batch” though so that we take manageable steps. Johan
Created attachment 768975 [details] mongodb.init-fix1.patch Fixes to the init start/stop script to make it behave correctly when shutting down a loaded database.
Created attachment 768976 [details] mongodb.spec-fix1.patch Cleanup and fixes to the spec-file. Renamed mongodb-devel to libmongodb-devel.
So, the compile problem on Fedora 19 is caused by how GCC 4.8 treat unused typedefs in some situations. I have created a patch that fixes this in the same way as suggested on a couple of places on the net, including the gcc bugtrack. I'm attaching the patch and a patch to the spec file to incorporate the gcc patch. I have successfully compiled the package with this patch on Fedora 17 i386/x86_64, Fedora 19 x86_64 and CentOS 6.4 x86_64 so it looks like the patch does not interfere with older compilers. Regards Johan
Created attachment 770102 [details] mongodb-2.4.4-gcc48.patch Patch to make mongodb compile with GCC 4.8
Created attachment 770103 [details] mongodb.spec-fix2.patch Patch the the spec file to add the GCC 4.8 patch.
I have gotten your patches and changes put together. They are here http://tdawson.fedorapeople.org/mongodb/mongodb-2.4.4-4.fc20.src.rpm http://tdawson.fedorapeople.org/mongodb/mongodb.2.4.4-4.spec Thank you so much for the spec file changes. Looking at some of them I'm surprised that nobody complained before. The only change I think I would make would be to make Obsoletes: mongodb-devel <= 2.2.4-2 to Obsoletes: mongodb-devel < 2.4 That would make the break at a 2.4, which is much cleaner. I am currently working on the update to 2.4.5. This needs to be done due to CVE-2013-4650 Not all of the patches are applying cleanly, but I should have it done by the end of the day.
Obsoletes: mongodb-devel < 2.4 sounds perfectly ok for me. Go ahead and change that! With 2.4.5 you should drop Patch4 and also check if you can drop Patch3 as well. Also, you will need to create a patch so that libmongoclient.so gets installed in /usr/lib64 on x86_64 since the install script in 2.4.5 installs it in /usr/lib for all architectures. See https://jira.mongodb.org/browse/SERVER-10049. I'm also quite sure that you can drop the code in the spec file that removes the extra install of header files since this should be fixed in 2.4.5.
http://tdawson.fedorapeople.org/mongodb/mongodb-2.4.5-1.fc20.src.rpm http://tdawson.fedorapeople.org/mongodb/mongodb.2.4.5-1.spec http://tdawson.fedorapeople.org/mongodb/mongodb.spec Yep, patches 3 and 4 could be dropped, which I did. I did a 'sed' instead of a patch for the /usr/lib problem. A patch isn't dynamic, so it can't tell where to put things during build time. I also commented out the fix for the extra header problem we had. I haven't had time for a full build of all of these, I've only verified that the patches and sed do what was expected of them. But I have them building now, we'll know in a while if everything is fixed and working.
Great! The sed approach was what I was thinking about for exactly the same reason :-) I'm in the process of breaking down the rest of my suggestions in manageable chunks and will post here as soon as possible.
Two comments/concerns. Have you tested the new init.d script on both single node mongo servers, as well as sharded servers? What is the reasoning for wanting to break out mongos? Can you do sharding without the server? I'm very concerned that this will start breaking users sharded machines when they do an update, and I don't see a reason for it.
I have worked with David Marlin and he got the arm patches fixed up and working. After much trial and error I found that the hardening breaks 2.4.x. I had the if statements messed up, so it really wasn't calling the hardening stuff. Tracking it down, it has to do with the s2 code. I have alot of people asking for mongodb 2.4.x in any form, and very few asking for the hardening stuff. I am leaving in the patch that allows us to do the hardening, but not calling the hardening in the spec file. In this format I have tested it across f18-f20, EPEL6, and on arm f18-f20. It works for all of those. I am going to commit this, just to rawhide. That will allow those that really want it to use it and test it. We need to find out how backwards compatible it is, and how easy/hard it will be to push it to F18,F19 and EPEL6. http://tdawson.fedorapeople.org/mongodb/mongodb-2.4.5-3.fc20.src.rpm http://tdawson.fedorapeople.org/mongodb/mongodb.2.4.5-3.spec http://tdawson.fedorapeople.org/mongodb/mongodb.spec Troy
Troy, Your changes and way forward sounds ok. Hardening is not as important for me (running mongodb on a separate network) so I leave that to others to argue for. As for your questions about mongos/sharding/mongod: I'm using mongodb in both stand alone, replica set and sharded environments with rpm:s that includes everything I'm suggesting here. So the changes are battle proven for the last three-four months, including the init.d script. Upgrade paths from a 2.2 package from EPEL/Fedora to my own packages is something that I have not addressed all the way though since I wasn't in need of it. The reason for wanting to break out mongos and put it in its own sub-package is that mongos is a service you normally only run on your app-servers. Bringing in the whole database on those servers is not necessary. In the same way there is no need for the mongos binary to be installed on every machine that runs mongod (not even when the mongod:s are part of a sharded setup). So it's a matter of getting just the right amount of files needed on a machine for it to do it's job. You are right though that this i a bit backwards breaking and needs to be addressed. Someone that have installed mongodb-server on their app-server just to get the mongos binary will perhaps be surprised to see the the mongos binary disappeared when doing an update. Do you know of any standard way/pattern to let RPM bring in a new package on an update without a need for a Requires-directive? I'm thinking if triggers could be used so that a user updating mongodb-server from 2.2 to 2.4 will get the new package for mongos installed as well, while a user installing mongodb-server 2.4 for the first time will not. To not get stuck with this issue, I suggest that we at least move on and bring in the necessary files in the mongodb-server package for making a proper service of mongos (init.d script, logrotate script, systemd service unit etc). Then we can see if a separate package for mongos is doable or not. Before bringing in the new files, I would like to bring up a suggestion for some name changes. This is also a bit backwards breaking, but makes things clearer in my opinion. As it is now, the following files: /etc/mongodb.conf /etc/logrotate.d/mongodb /var/run/mongodb/mongodb.pid /var/log/mongodb/mongodb.log all have the name mongodb.* even though they relate to the binary/service mongod. When bringing in mongos in the game it makes more sense to rename all these files to mongod.* instead. I realize that a name change on /etc/mongodb.conf is perhaps a bit to much to ask for since it's probably confusing for users upgrading from 2.2 but what do you think about the others? With such a change we would have: /etc/mongodb.conf /etc/logrotate.d/mongod /var/run/mongodb/mongod.pid /var/log/mongodb/mongod.log for mongod. For mongos I'm suggest that we run the mongos service as the mongodb user and re-use the base directories from mongod for mongos. Thus, for mongos we will have the following new files added to the package: /etc/mongos.conf /etc/logrotate.d/mongos /var/run/mongodb/mongos.pid /var/log/mongodb/mongos.log /usr/lib/systemd/system/mongos.service The current logrotate script for mongod rotates all files in /var/log/mongodb so that has to be changed so that /etc/logrotate.d/mongod only rotates /var/log/mongodb/mongod.log and the new /etc/logrotate.d/mongos only rotates /var/log/mongodb/mongod.log. /etc/logrotate.d/mongodb is marked as %config(noreplace) so changing that file could possibly have side effects for users that have changed the file themselves. Regardless of the above, the current logrotate script does not clean up /var/log/mongodb properly since mongod itself rename files when rotating them in a way that the logrotate script does not handle. So fixing this require changes to the current logrotate script anyhow. Finally, I'm suggesting that we drop /etc/sysconfig/mongod. mongod does not have any special command line arguments. Anything you could put on the command line could be put in the config file. Having two places to put config that does the same thing is confusing. The current /etc/init.d/mongod script is even written i such way that it does not make any use of /etc/sysconfig/mongod. A lot of changes suggested in this post, but I was not able to break them down in manageable chunks since many of them relate to each other. I hope that you are able to at least grab some of them and I'm looking forward to your comments. I have all the files needed for making mongos a service ready to post here as soon as we agree on how to proceed. Johan
mongodb-2.4.6-1.fc19 has been submitted as an update for Fedora 19. https://admin.fedoraproject.org/updates/mongodb-2.4.6-1.fc19
Package mongodb-2.4.6-1.fc19: * should fix your issue, * was pushed to the Fedora 19 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing mongodb-2.4.6-1.fc19' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2013-17283/mongodb-2.4.6-1.fc19 then log in and leave karma (feedback).
mongodb-2.4.6-1.fc19 has been pushed to the Fedora 19 stable repository. If problems still persist, please make note of it in this bug report.