Description of problem: Pulp database is migrated before pulp-plugins-docker is installed when upgrading a 6.0 capsule, resulting in these errors in the logs: Importer docker_importer: no type definition found for docker_image Jun 05 08:59:09 capsule.stbenjam.satellite.lab.eng.rdu2.redhat.com pulp[15120]: pulp.server.webservices.application:CRITICAL: ************************************************************* Version-Release number of selected component (if applicable): 6.1 SNAP 7 How reproducible: Always Steps to Reproduce: 1. Install 6.0 Capsule 2. Upgrade to 6.1 Actual results: Errors in logs Expected results: No errors in logs Additional info: For upgrade scenario, we should probably always run pulp migration again to ensure any additional plugins installed get their migrations done
Since this issue was entered in Red Hat Bugzilla, the release flag has been set to ? to ensure that it is properly evaluated for this release.
Created redmine issue http://projects.theforeman.org/issues/10716 from this bug
*** Bug 1228221 has been marked as a duplicate of this bug. ***
This issue is also observed in GA Snap 8, logs in /var/log/messages, Jun 12 05:35:43 sat6-capsule-rhel71 pulp: pulp.server.initialization:ERROR: One or more plugins failed to initialize. If a new type has been added, run pulp-manage-db to load the type into the database and restart the application. Error message: Importer docker_importer: no type definition found for docker_image
WORKAROUND: capsule-installer --upgrade twice
Workaround works: [root@sat6-capsule-rhel66 ~]# capsule-installer --upgrade --certs-tar sat6-capsule-rhel66.usersys.redhat.com-certs.tar Upgrading... Upgrade Step: stop_services... Upgrade Step: start_mongo... Upgrade Step: migrate_pulp... Upgrade Step: Running installer... Installing Done [100%] [..................................................................] The full log is at /var/log/capsule-installer/capsule-installer.log Katello upgrade completed!
The workaround mentioned in comment6 works and installer succeed. But when we ran capsule sync, it hanged at 75%. So I think following services needs a restart for i in qpidd pulp_workers pulp_celerybeat pulp_resource_manager httpd; do service $i restart; done
Moving to POST since upstream bug http://projects.theforeman.org/issues/10716 has been closed ------------- Anonymous Applied in changeset commit:katello-installer|63ffb72c46af790bd6d7f4908b8f8796abd1dc84.
Tested this on latest Sat GA snap 11 Compose 2. Currently the satellite upgrade is failing, bug [1]. And my capsule is registered with satellite before upgrade. And hence after upgrading the satellite to Sat 6.1(which is failing with NameError), capsule upgrade is also failing with the same NameError. [1] : https://bugzilla.redhat.com/show_bug.cgi?id=1239020 : satellite upgrade is failing from sat 6.0.8 to 6.1.1 GA snap 11
If we apply the patch from bz 1239020 then the reported issue is no longer reproducible. But we will re-test this with next snap with fix.
Reproducible with snap13 on rhel71. Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.db.migrate.models:ERROR: pulp.server.db.migrations hasn't been updated to the latest available migration. Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.db.migrate.models:ERROR: pulp_rpm.plugins.migrations hasn't been updated to the latest available migration. Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:CRITICAL: ************************************************************* Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:CRITICAL: The Pulp server failed to start due to the following reasons: Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896) The database has not been migrated to the current version. Run pulp-manage-db and restart the application. Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896) Traceback (most recent call last): Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 147, in wsgi_application Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896) _initialize_pulp() Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 102, in _initialize_pulp Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896) migration_models.check_package_versions() Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896) File "/usr/lib/python2.7/site-packages/pulp/server/db/migrate/models.py", line 291, in check_package_versions Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896) raise Exception(error_message) Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896) InitializationException: The database has not been migrated to the current version. Run pulp-manage-db and restart the application. Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:CRITICAL: ************************************************************* Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.initialization:ERROR: One or more plugins failed to initialize. If a new type has been added, run pulp-manage-db to load the type into the database and restart the application. Error message: Importer docker_importer: no type definition found for docker_image Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:CRITICAL: ************************************************************* Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) The Pulp server encountered an unexpected failure during initialization Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) Traceback (most recent call last): Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 147, in wsgi_application Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) _initialize_pulp() Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 94, in _initialize_pulp Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) from pulp.server.async import app Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) File "/usr/lib/python2.7/site-packages/pulp/server/async/app.py", line 12, in <module> Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) initialization.initialize() Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) File "/usr/lib/python2.7/site-packages/pulp/server/initialization.py", line 35, in initialize Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) plugin_api.initialize() Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) File "/usr/lib/python2.7/site-packages/pulp/plugins/loader/api.py", line 79, in initialize Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) _validate_importers() Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) File "/usr/lib/python2.7/site-packages/pulp/plugins/loader/api.py", line 498, in _validate_importers Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) raise loader_exceptions.InvalidImporter(msg %% {'i': plugin_id, 't': type_}) Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) InvalidImporter: Importer docker_importer: no type definition found for docker_image Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:CRITICAL: ************************************************************* Jul 16 04:25:02 cloud-q
I'm not sure we can fix this for 6.1.1. Upstream, the puppet-pulp modules are undergoing a major refactoring so that we will migrate the database correctly before starting the services, but the entire structure of the puppet module is changing, and it's not compatible with 6.1 at all. The way things in the installer happen now is: 1. Stop services except mongo 2. Migrate Pulp database 3. Install pulp-docker-plugins 4. Pulp configuration, if any. Start pulp services 5. ERROR because docker_image isn't in database 6. Migrate Pulp database (docker_image is added) 7. Service restart 8. Pulp is OK Step 8 where pulp is OK happens shortly after your errors: Jul 16 04:25:17 cloud-qe-22.idmqe.lab.eng.bos.redhat.com pulp[24877]: pulp.plugins.loader.manager:INFO: Loaded plugin docker_importer for types: docker_image Jul 16 04:25:17 cloud-qe-22.idmqe.lab.eng.bos.redhat.com pulp[24877]: pulp.server.webservices.application:INFO: The Pulp server has been successfully initialized So, pulp does correct itself on your capsule. Can you verify the capsule is working correctly after the upgrade is complete? Maybe initiate a sync. It should be OK.
merged.
*** Bug 1244690 has been marked as a duplicate of this bug. ***
Sorry, this will fail QA. *sigh* I hate this bug. There's about 6 things that happen that have to occur in a very exact order, and I'm doing this largely blind because the problem isn't upstream. We've had docker_image for 3 releases. Anyway, the last thing will be to reverse the order of httpd and migrating pulp. I've tested that on an upgrade with SNAP 14. https://github.com/Katello/katello-installer/pull/246 Incidentally, migrating before starting HTTP fixes https://bugzilla.redhat.com/show_bug.cgi?id=1244808, so at least there's a nice 2-for-1 here.
Thanks Stephen. Moving this back to assigned state as per comment 19.
Upstream change is now merged, so hopefully this can make the next snap: katello-installer|8c61f9ce7b10c48b07de7ef02b1f49ce28e95143 If you do want to try it anyway, it's a simple change (see the linked PR in comment #19). I've tried it myself now and I do get a fully clean install, no errors. Sorry for all the churn on this one.
Verified this bug with latest Satellite/Capsule GA snap 15 latest. By upgrading the Sat 6.0.8 to Sat 6.1 GA. The issue is no more reproducible and seems it is fixed. Hence closing this as Verified.
This bug was fixed in Satellite 6.1.1 which was delivered on 12 August, 2015.