I think there's an issue with how our logs.py sets up logging. It has a start and stop function, using a global variable to prevent multiple starts without a stop in between. The problem is, the stop doesn't properly clean up the configuration that's done in the start method. So each time it's stopped/started, we add all the handlers back onto the loggers. This shouldn't be a problem (currently) in the running code since it's only started in the application bootstrap. However our unit tests are badly broken. I noticed in the debugger that there were a ton of duplicate handlers on loggers when running tests. The stop/start process is tied to each time the configuration is loaded in a test, which is done in setUp. Since that's called once for each test, it's causing our logging configuration to blow up by the end of the test runs. As of right now, the test_consumer_history_api tests hang for both me and John Matthews. They run fine when I comment out the testutil.start_logging() call. This might also explain why Hudson is hung right now (current build has been going for 15h). Python seems to get especially confused in the auditing logging configuration since it's adding a bunch of TimedRotatingFileHandlers. I'm curious if it I'll file a high priority bug for it since it's stopping our tests and builds from running. I'm not 100% sure of the fix, but the issue looks to be that our stop/start logging calls are bugged.
Created attachment 446517 [details] Debugger view of the log on the first test
Created attachment 446518 [details] Debugger view of the log after multiple tests
Lowered to medium since the tests can at least run again.
FYI, in the two attached screenshots, look at the handlers list under logging in the bottom half. I forgot to draw a circle around the relevant part of those images.
added some code to clear python's logging module of existing loggers when stop_logging is called
Fixed in 0.77.
Fixed in build 0.78.
Looks like this has not been an issue for quite sometime. moving to verified.
Closing with Community Release 15 pulp-0.0.223-4.