Bug 1228694
Summary: | Capsule upgrade fails with unknown image type docker | ||
---|---|---|---|
Product: | Red Hat Satellite | Reporter: | Stephen Benjamin <stbenjam> |
Component: | Upgrades | Assignee: | Stephen Benjamin <stbenjam> |
Status: | CLOSED CURRENTRELEASE | QA Contact: | Jitendra Yejare <jyejare> |
Severity: | high | Docs Contact: | |
Priority: | unspecified | ||
Version: | 6.1.0 | CC: | bbuckingham, bkearney, chpeters, cwelton, jyejare, mmccune, sghai |
Target Milestone: | Unspecified | Keywords: | Triaged |
Target Release: | Unused | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
URL: | http://projects.theforeman.org/issues/10716 | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2015-08-12 16:03:47 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 1195450, 1244808 |
Description
Stephen Benjamin
2015-06-05 13:06:02 UTC
Since this issue was entered in Red Hat Bugzilla, the release flag has been set to ? to ensure that it is properly evaluated for this release. Created redmine issue http://projects.theforeman.org/issues/10716 from this bug *** Bug 1228221 has been marked as a duplicate of this bug. *** This issue is also observed in GA Snap 8, logs in /var/log/messages, Jun 12 05:35:43 sat6-capsule-rhel71 pulp: pulp.server.initialization:ERROR: One or more plugins failed to initialize. If a new type has been added, run pulp-manage-db to load the type into the database and restart the application. Error message: Importer docker_importer: no type definition found for docker_image WORKAROUND: capsule-installer --upgrade twice Workaround works: [root@sat6-capsule-rhel66 ~]# capsule-installer --upgrade --certs-tar sat6-capsule-rhel66.usersys.redhat.com-certs.tar Upgrading... Upgrade Step: stop_services... Upgrade Step: start_mongo... Upgrade Step: migrate_pulp... Upgrade Step: Running installer... Installing Done [100%] [..................................................................] The full log is at /var/log/capsule-installer/capsule-installer.log Katello upgrade completed! The workaround mentioned in comment6 works and installer succeed. But when we ran capsule sync, it hanged at 75%. So I think following services needs a restart for i in qpidd pulp_workers pulp_celerybeat pulp_resource_manager httpd; do service $i restart; done Moving to POST since upstream bug http://projects.theforeman.org/issues/10716 has been closed ------------- Anonymous Applied in changeset commit:katello-installer|63ffb72c46af790bd6d7f4908b8f8796abd1dc84. Tested this on latest Sat GA snap 11 Compose 2. Currently the satellite upgrade is failing, bug [1]. And my capsule is registered with satellite before upgrade. And hence after upgrading the satellite to Sat 6.1(which is failing with NameError), capsule upgrade is also failing with the same NameError. [1] : https://bugzilla.redhat.com/show_bug.cgi?id=1239020 : satellite upgrade is failing from sat 6.0.8 to 6.1.1 GA snap 11 If we apply the patch from bz 1239020 then the reported issue is no longer reproducible. But we will re-test this with next snap with fix. Reproducible with snap13 on rhel71. Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.db.migrate.models:ERROR: pulp.server.db.migrations hasn't been updated to the latest available migration. Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.db.migrate.models:ERROR: pulp_rpm.plugins.migrations hasn't been updated to the latest available migration. Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:CRITICAL: ************************************************************* Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:CRITICAL: The Pulp server failed to start due to the following reasons: Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896) The database has not been migrated to the current version. Run pulp-manage-db and restart the application. Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896) Traceback (most recent call last): Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 147, in wsgi_application Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896) _initialize_pulp() Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 102, in _initialize_pulp Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896) migration_models.check_package_versions() Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896) File "/usr/lib/python2.7/site-packages/pulp/server/db/migrate/models.py", line 291, in check_package_versions Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896) raise Exception(error_message) Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896) InitializationException: The database has not been migrated to the current version. Run pulp-manage-db and restart the application. Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:CRITICAL: ************************************************************* Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.initialization:ERROR: One or more plugins failed to initialize. If a new type has been added, run pulp-manage-db to load the type into the database and restart the application. Error message: Importer docker_importer: no type definition found for docker_image Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:CRITICAL: ************************************************************* Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) The Pulp server encountered an unexpected failure during initialization Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) Traceback (most recent call last): Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 147, in wsgi_application Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) _initialize_pulp() Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 94, in _initialize_pulp Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) from pulp.server.async import app Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) File "/usr/lib/python2.7/site-packages/pulp/server/async/app.py", line 12, in <module> Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) initialization.initialize() Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) File "/usr/lib/python2.7/site-packages/pulp/server/initialization.py", line 35, in initialize Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) plugin_api.initialize() Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) File "/usr/lib/python2.7/site-packages/pulp/plugins/loader/api.py", line 79, in initialize Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) _validate_importers() Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) File "/usr/lib/python2.7/site-packages/pulp/plugins/loader/api.py", line 498, in _validate_importers Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) raise loader_exceptions.InvalidImporter(msg %% {'i': plugin_id, 't': type_}) Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) InvalidImporter: Importer docker_importer: no type definition found for docker_image Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:CRITICAL: ************************************************************* Jul 16 04:25:02 cloud-q I'm not sure we can fix this for 6.1.1. Upstream, the puppet-pulp modules are undergoing a major refactoring so that we will migrate the database correctly before starting the services, but the entire structure of the puppet module is changing, and it's not compatible with 6.1 at all. The way things in the installer happen now is: 1. Stop services except mongo 2. Migrate Pulp database 3. Install pulp-docker-plugins 4. Pulp configuration, if any. Start pulp services 5. ERROR because docker_image isn't in database 6. Migrate Pulp database (docker_image is added) 7. Service restart 8. Pulp is OK Step 8 where pulp is OK happens shortly after your errors: Jul 16 04:25:17 cloud-qe-22.idmqe.lab.eng.bos.redhat.com pulp[24877]: pulp.plugins.loader.manager:INFO: Loaded plugin docker_importer for types: docker_image Jul 16 04:25:17 cloud-qe-22.idmqe.lab.eng.bos.redhat.com pulp[24877]: pulp.server.webservices.application:INFO: The Pulp server has been successfully initialized So, pulp does correct itself on your capsule. Can you verify the capsule is working correctly after the upgrade is complete? Maybe initiate a sync. It should be OK. merged. *** Bug 1244690 has been marked as a duplicate of this bug. *** Sorry, this will fail QA. *sigh* I hate this bug. There's about 6 things that happen that have to occur in a very exact order, and I'm doing this largely blind because the problem isn't upstream. We've had docker_image for 3 releases. Anyway, the last thing will be to reverse the order of httpd and migrating pulp. I've tested that on an upgrade with SNAP 14. https://github.com/Katello/katello-installer/pull/246 Incidentally, migrating before starting HTTP fixes https://bugzilla.redhat.com/show_bug.cgi?id=1244808, so at least there's a nice 2-for-1 here. Thanks Stephen. Moving this back to assigned state as per comment 19. Upstream change is now merged, so hopefully this can make the next snap: katello-installer|8c61f9ce7b10c48b07de7ef02b1f49ce28e95143 If you do want to try it anyway, it's a simple change (see the linked PR in comment #19). I've tried it myself now and I do get a fully clean install, no errors. Sorry for all the churn on this one. Verified this bug with latest Satellite/Capsule GA snap 15 latest. By upgrading the Sat 6.0.8 to Sat 6.1 GA. The issue is no more reproducible and seems it is fixed. Hence closing this as Verified. This bug was fixed in Satellite 6.1.1 which was delivered on 12 August, 2015. |