Bug 1228694 - Capsule upgrade fails with unknown image type docker
Summary: Capsule upgrade fails with unknown image type docker
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Upgrades
Version: 6.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: Unspecified
Assignee: Stephen Benjamin
QA Contact: Jitendra Yejare
URL: http://projects.theforeman.org/issues...
Whiteboard:
: 1228221 (view as bug list)
Depends On:
Blocks: 1195450 1244808
TreeView+ depends on / blocked
 
Reported: 2015-06-05 13:06 UTC by Stephen Benjamin
Modified: 2019-07-11 09:19 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-08-12 16:03:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Stephen Benjamin 2015-06-05 13:06:02 UTC
Description of problem:
Pulp database is migrated before pulp-plugins-docker is installed when upgrading a 6.0 capsule, resulting in these errors in the logs:

Importer docker_importer: no type definition found for docker_image
Jun 05 08:59:09 capsule.stbenjam.satellite.lab.eng.rdu2.redhat.com pulp[15120]: pulp.server.webservices.application:CRITICAL: *************************************************************


Version-Release number of selected component (if applicable):
6.1 SNAP 7

How reproducible:
Always

Steps to Reproduce:
1. Install 6.0 Capsule
2. Upgrade to 6.1

Actual results:
Errors in logs

Expected results:
No errors in logs

Additional info:
For upgrade scenario, we should probably always run pulp migration again to ensure any additional plugins installed get their migrations done

Comment 1 RHEL Program Management 2015-06-05 13:12:48 UTC
Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

Comment 3 Stephen Benjamin 2015-06-05 14:07:20 UTC
Created redmine issue http://projects.theforeman.org/issues/10716 from this bug

Comment 4 Stephen Benjamin 2015-06-05 18:29:52 UTC
*** Bug 1228221 has been marked as a duplicate of this bug. ***

Comment 5 Jitendra Yejare 2015-06-12 10:21:20 UTC
This issue is also observed in GA Snap 8, logs in /var/log/messages,

Jun 12 05:35:43 sat6-capsule-rhel71 pulp: pulp.server.initialization:ERROR: One or more plugins failed to initialize. If a new type has been added, run pulp-manage-db to load the type into the database and restart the application. Error message: Importer docker_importer: no type definition found for docker_image

Comment 6 Mike McCune 2015-06-15 16:03:58 UTC
WORKAROUND:

capsule-installer --upgrade twice

Comment 7 Sachin Ghai 2015-06-15 16:26:49 UTC
Workaround works:

[root@sat6-capsule-rhel66 ~]# capsule-installer --upgrade --certs-tar sat6-capsule-rhel66.usersys.redhat.com-certs.tar
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_mongo...
Upgrade Step: migrate_pulp...
Upgrade Step: Running installer...
Installing             Done                                               [100%] [..................................................................]
  The full log is at /var/log/capsule-installer/capsule-installer.log
Katello upgrade completed!

Comment 8 Sachin Ghai 2015-06-17 10:45:53 UTC
The workaround mentioned in comment6 works and installer succeed. But when we ran capsule sync, it hanged at 75%. So I think following services needs a restart

 for i in qpidd pulp_workers pulp_celerybeat pulp_resource_manager httpd; do service $i restart; done

Comment 9 Bryan Kearney 2015-06-29 16:05:23 UTC
Moving to POST since upstream bug http://projects.theforeman.org/issues/10716 has been closed
-------------
Anonymous
Applied in changeset commit:katello-installer|63ffb72c46af790bd6d7f4908b8f8796abd1dc84.

Comment 11 Jitendra Yejare 2015-07-08 08:52:01 UTC
Tested this on latest Sat GA snap 11 Compose 2.

Currently the satellite upgrade is failing, bug [1]. And my capsule is registered with satellite before upgrade.

And hence after upgrading the satellite to Sat 6.1(which is failing with NameError), capsule upgrade is also failing with the same NameError.

[1] : https://bugzilla.redhat.com/show_bug.cgi?id=1239020 : satellite upgrade is failing from sat 6.0.8 to 6.1.1 GA snap 11

Comment 12 Sachin Ghai 2015-07-14 06:20:40 UTC
If we apply the patch from bz 1239020 then the reported issue is no longer reproducible. But we will re-test this with next snap with fix.

Comment 13 Sachin Ghai 2015-07-16 08:35:58 UTC
Reproducible with snap13 on rhel71.

Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.db.migrate.models:ERROR: pulp.server.db.migrations hasn't been updated to the latest available migration.
Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.db.migrate.models:ERROR: pulp_rpm.plugins.migrations hasn't been updated to the latest available migration.
Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:CRITICAL: *************************************************************
Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:CRITICAL: The Pulp server failed to start due to the following reasons:
Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896)   The database has not been migrated to the current version. Run pulp-manage-db and restart the application.
Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896) Traceback (most recent call last):
Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 147, in wsgi_application
Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896)     _initialize_pulp()
Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 102, in _initialize_pulp
Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896)     migration_models.check_package_versions()
Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896)   File "/usr/lib/python2.7/site-packages/pulp/server/db/migrate/models.py", line 291, in check_package_versions
Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896)     raise Exception(error_message)
Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (22406-40896) InitializationException: The database has not been migrated to the current version. Run pulp-manage-db and restart the application.
Jul 16 04:24:01 cloud-qe-22 pulp: pulp.server.webservices.application:CRITICAL: *************************************************************


Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.initialization:ERROR: One or more plugins failed to initialize. If a new type has been added, run pulp-manage-db to load the type into the database and restart the application. Error message: Importer docker_importer: no type definition found for docker_image
Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:CRITICAL: *************************************************************
Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) The Pulp server encountered an unexpected failure during initialization
Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) Traceback (most recent call last):
Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 147, in wsgi_application
Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672)     _initialize_pulp()
Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 94, in _initialize_pulp
Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672)     from pulp.server.async import app
Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672)   File "/usr/lib/python2.7/site-packages/pulp/server/async/app.py", line 12, in <module>
Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672)     initialization.initialize()
Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672)   File "/usr/lib/python2.7/site-packages/pulp/server/initialization.py", line 35, in initialize
Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672)     plugin_api.initialize()
Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672)   File "/usr/lib/python2.7/site-packages/pulp/plugins/loader/api.py", line 79, in initialize
Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672)     _validate_importers()
Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672)   File "/usr/lib/python2.7/site-packages/pulp/plugins/loader/api.py", line 498, in _validate_importers
Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672)     raise loader_exceptions.InvalidImporter(msg %% {'i': plugin_id, 't': type_})
Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:ERROR: (24267-24672) InvalidImporter: Importer docker_importer: no type definition found for docker_image
Jul 16 04:25:02 cloud-qe-22 pulp: pulp.server.webservices.application:CRITICAL: *************************************************************
Jul 16 04:25:02 cloud-q

Comment 14 Stephen Benjamin 2015-07-16 13:54:52 UTC
I'm not sure we can fix this for 6.1.1.  

Upstream, the puppet-pulp modules are undergoing a major refactoring so that we will migrate the database correctly before starting the services, but the entire structure of the puppet module is changing, and it's not compatible with 6.1 at all.

The way things in the installer happen now is:

1. Stop services except mongo
2. Migrate Pulp database
3. Install pulp-docker-plugins
4. Pulp configuration, if any.  Start pulp services
5. ERROR because docker_image isn't in database
6. Migrate Pulp database (docker_image is added)
7. Service restart
8. Pulp is OK

Step 8 where pulp is OK happens shortly after your errors:

Jul 16 04:25:17 cloud-qe-22.idmqe.lab.eng.bos.redhat.com pulp[24877]: pulp.plugins.loader.manager:INFO: 
Loaded plugin docker_importer for types: docker_image
Jul 16 04:25:17 cloud-qe-22.idmqe.lab.eng.bos.redhat.com pulp[24877]: pulp.server.webservices.application:INFO: The Pulp server has been successfully initialized


So, pulp does correct itself on your capsule.  Can you verify the capsule is working correctly after the upgrade is complete? Maybe initiate a sync.  It should be OK.

Comment 16 Bryan Kearney 2015-07-21 16:18:08 UTC
merged.

Comment 17 Stephen Benjamin 2015-07-22 10:41:46 UTC
*** Bug 1244690 has been marked as a duplicate of this bug. ***

Comment 19 Stephen Benjamin 2015-07-22 12:45:14 UTC
Sorry, this will fail QA. *sigh*  I hate this bug.

There's about 6 things that happen that have to occur in a very exact order, and I'm doing this largely blind because the problem isn't upstream. We've had docker_image for 3 releases.

Anyway, the last thing will be to reverse the order of httpd and migrating pulp.
I've tested that on an upgrade with SNAP 14.
  https://github.com/Katello/katello-installer/pull/246

Incidentally, migrating before starting HTTP fixes https://bugzilla.redhat.com/show_bug.cgi?id=1244808, so at least there's a nice 2-for-1 here.

Comment 20 Sachin Ghai 2015-07-22 13:33:06 UTC
Thanks Stephen. Moving this back to assigned state as per comment 19.

Comment 21 Stephen Benjamin 2015-07-22 14:33:45 UTC
Upstream change is now merged, so hopefully this can make the next snap:

   katello-installer|8c61f9ce7b10c48b07de7ef02b1f49ce28e95143

If you do want to try it anyway, it's a simple change (see the linked PR in comment #19).  I've tried it myself now and I do get a fully clean install, no errors. Sorry for all the churn on this one.

Comment 23 Jitendra Yejare 2015-07-30 12:38:22 UTC
Verified this bug with latest Satellite/Capsule GA snap 15 latest. By upgrading the Sat 6.0.8 to Sat 6.1 GA.

The issue is no more reproducible and seems it is fixed.

Hence closing this as Verified.

Comment 24 Bryan Kearney 2015-08-12 16:03:47 UTC
This bug was fixed in Satellite 6.1.1 which was delivered on 12 August, 2015.


Note You need to log in before you can comment on or make changes to this bug.