Bug 885261 - katello-upgrade fails from 1.0 to 1.1
katello-upgrade fails from 1.0 to 1.1
Status: CLOSED UPSTREAM
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Installer (Show other bugs)
6.0.0
x86_64 Linux
unspecified Severity high (vote)
: Unspecified
: --
Assigned To: Lukas Zapletal
Katello QA List
: Triaged
Depends On:
Blocks: 886142
  Show dependency treegraph
 
Reported: 2012-12-07 18:38 EST by Ben Breard
Modified: 2014-01-27 09:12 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 886142 (view as bug list)
Environment:
Last Closed: 2013-09-19 14:13:01 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
katello-debug attached (6.45 MB, application/gzip)
2012-12-07 18:38 EST, Ben Breard
no flags Details

  None (edit)
Description Ben Breard 2012-12-07 18:38:05 EST
Created attachment 659647 [details]
katello-debug attached

Description of problem:
Upgrading from 1.0 to 1.1 fails during the 0004_migrate_katello_sb.sh:
3/4: Migrate Katello database (0004_migrate_katello_db.sh)
 Updates Katello database schema to the latest version    

Do you want to proceed? (y/n): y

Migrate Katello database FAILED.

Looking at the database there are a lot of "Red Hat" providers and only 3 orgs. I suspect this was caused from having several manifests that failed to import. That seems to be killing the upgrade process.



Version-Release number of selected component (if applicable):
We were running an up to date version of 1.0 before the upgrade to 1.1 GA.

How reproducible:
At this point, very

Steps to Reproduce:
From a clean install, I'm not too sure. Heavily use the system and then try to upgrade???
Comment 2 Lukas Zapletal 2012-12-10 10:28:01 EST
Ben,

thanks for the report, going to reproduce. Do you still have manifests that failed? I will try with simple incorrect ZIP file first.
Comment 4 Lukas Zapletal 2012-12-11 10:10:29 EST
Ben,

after investigation of your logs, I can see you have broken installation. From the history I can see:

1) You have installed on 09/19 with "Red_Hat" default organization
2) You have re-run katello-configure on 09/25 (for what reason?)
3) You have initiated upgrade which broke the installation

Now, the issue is in the step number two - you have re-run katello-configure three times. This does not hurt, but you need to provide it answer file every time:

# katello-configure --answer-file=/etc/katello/katello-configure.conf

but you apparently did this:

# katello-configure

which cleared the default organization setting ("Red_Hat") to the default one ("ACME_Corporation"). Then the upgrade broke the setup because it was trying (or created) ACME_Corporation with default Red Hat provider.

I need to see your database to recommend you what you can do with your setup. But I wonder the only possible recovery will be to delete organizations, providers and start over. Synchronization should be fast as the data will remain on the hard disk. If this does not work for you, you can completely reset your database using the following command:

# katello-configure --reset-data=YES

WARNING: This will completely delete all the data (providers, systems etc). Packages are not deleted, so after import synchronization should be fast.

You can do this and you will no longer need to finish with upgrade script because this re-initialize everything including (broken) search index.

I think you have pointed out on a dangerous thing - rerunning katello-configure without --answer-file can break installation and we should consider to change it's behavior to automatically load answer file. I have to admit this beavior is just wrong and I will start discussion on changing this ASAP.
Comment 6 Ben Breard 2012-12-11 12:14:37 EST
I'm afraid I'm not going to be able to help much here. This is a shared environment that many people have had their hands on. Unfortunately I can't provide the debug-certificates, or a database dump as we had to reinstall the system to stand up 1.1. I also don't know the reason katello-configure was run multiple times. I'm sorry this isn't very helpful. :/
Comment 7 Lukas Zapletal 2012-12-12 03:50:23 EST
Ben,

please do --reset-data=YES to fully reinitialize instance and re-import all manifests and sync the content.

I am going to do the change into katello-configure so it does not overwrite initialization options anymore.
Comment 8 Lukas Zapletal 2012-12-13 07:24:20 EST
So I am working on two fixes:

A) Make sure katello-configure always loads answer file

B) Make sure all providers and all the content of the providers are deleted when an organization is deleted (including Red Hat provider).

Note to QA: Testing A is straightforward, to test b create an org, sync a content into both Red Hat and a custom provider, promote it. Then delete the org and make sure all content was deleted (from all systems - katello database, candlepin, pulp).
Comment 9 Lukas Zapletal 2012-12-18 12:20:05 EST
So I am going implement two fixes. The first one is here:

https://github.com/Katello/katello/pull/1259

We are still testing it. The second will follow shortly (k-c answer file fix).
Comment 10 Lukas Zapletal 2012-12-21 08:08:59 EST
Ok the second fix is in regard katello-configure:

https://github.com/Katello/katello/pull/1330
Comment 11 Lukas Zapletal 2012-12-21 08:19:19 EST
Doco team: we will need this Errata text:

Due to error in the organization deletion code which runs in the background, red hat provider associated to the organization was not deleted from database. This may cause several issues when working with the system or upgrading it.

If you have deleted at least one organization, it is recommended to manually delete orhaned providers using the following script (please backup database first):

  /usr/share/katello/install/upgrade-scripts/0701_delete_undeleted_providers.rb

This script will be also automatically executed during upgrade to the next version.
Comment 14 Mike McCune 2013-09-19 14:13:01 EDT
These bugs have been resolved in upstream projects for a period of months so I'm mass-closing them as CLOSED:UPSTREAM.  If this is a mistake feel free to re-open.

Note You need to log in before you can comment on or make changes to this bug.