Ben,

thanks for the report, going to reproduce. Do you still have manifests that failed? I will try with simple incorrect ZIP file first.

Ben,

after investigation of your logs, I can see you have broken installation. From the history I can see:

1) You have installed on 09/19 with "Red_Hat" default organization
2) You have re-run katello-configure on 09/25 (for what reason?)
3) You have initiated upgrade which broke the installation

Now, the issue is in the step number two - you have re-run katello-configure three times. This does not hurt, but you need to provide it answer file every time:

# katello-configure --answer-file=/etc/katello/katello-configure.conf

but you apparently did this:

# katello-configure

which cleared the default organization setting ("Red_Hat") to the default one ("ACME_Corporation"). Then the upgrade broke the setup because it was trying (or created) ACME_Corporation with default Red Hat provider.

I need to see your database to recommend you what you can do with your setup. But I wonder the only possible recovery will be to delete organizations, providers and start over. Synchronization should be fast as the data will remain on the hard disk. If this does not work for you, you can completely reset your database using the following command:

# katello-configure --reset-data=YES

WARNING: This will completely delete all the data (providers, systems etc). Packages are not deleted, so after import synchronization should be fast.

You can do this and you will no longer need to finish with upgrade script because this re-initialize everything including (broken) search index.

I think you have pointed out on a dangerous thing - rerunning katello-configure without --answer-file can break installation and we should consider to change it's behavior to automatically load answer file. I have to admit this beavior is just wrong and I will start discussion on changing this ASAP.

I'm afraid I'm not going to be able to help much here. This is a shared environment that many people have had their hands on. Unfortunately I can't provide the debug-certificates, or a database dump as we had to reinstall the system to stand up 1.1. I also don't know the reason katello-configure was run multiple times. I'm sorry this isn't very helpful. :/

Ben,

please do --reset-data=YES to fully reinitialize instance and re-import all manifests and sync the content.

I am going to do the change into katello-configure so it does not overwrite initialization options anymore.

So I am working on two fixes:

A) Make sure katello-configure always loads answer file

B) Make sure all providers and all the content of the providers are deleted when an organization is deleted (including Red Hat provider).

Note to QA: Testing A is straightforward, to test b create an org, sync a content into both Red Hat and a custom provider, promote it. Then delete the org and make sure all content was deleted (from all systems - katello database, candlepin, pulp).

So I am going implement two fixes. The first one is here:

https://github.com/Katello/katello/pull/1259

We are still testing it. The second will follow shortly (k-c answer file fix).

Ok the second fix is in regard katello-configure:

https://github.com/Katello/katello/pull/1330

Doco team: we will need this Errata text:

Due to error in the organization deletion code which runs in the background, red hat provider associated to the organization was not deleted from database. This may cause several issues when working with the system or upgrading it.

If you have deleted at least one organization, it is recommended to manually delete orhaned providers using the following script (please backup database first):

  /usr/share/katello/install/upgrade-scripts/0701_delete_undeleted_providers.rb

This script will be also automatically executed during upgrade to the next version.

These bugs have been resolved in upstream projects for a period of months so I'm mass-closing them as CLOSED:UPSTREAM.  If this is a mistake feel free to re-open.