Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1388201 - candlepin content_id changes cause manfiest import/refresh failure
Summary: candlepin content_id changes cause manfiest import/refresh failure
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Candlepin
Version: 6.3.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: Unspecified
Assignee: Barnaby Court
QA Contact: Sanket Jagtap
URL:
Whiteboard:
Depends On: 1388234
Blocks: CEE_Sat6_Top_BZs, GSS_Sat6_Top_Bugs 1388551 1393442
TreeView+ depends on / blocked
 
Reported: 2016-10-24 17:54 UTC by sthirugn@redhat.com
Modified: 2021-12-10 14:46 UTC (History)
20 users (show)

Fixed In Version: candlepin-0.9.54.13-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1388234 1388236 1388241 1388551 1393439 1393442 1393444 (view as bug list)
Environment:
Last Closed: 2016-11-10 08:14:48 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
production.log (12.16 KB, text/plain)
2016-10-24 17:55 UTC, sthirugn@redhat.com 		no flags Details
candlepin error.log (46.80 KB, text/plain)
2016-10-24 17:55 UTC, sthirugn@redhat.com 		no flags Details
Default org manifest upload (93.29 KB, image/png)
2016-11-02 09:20 UTC, Sanket Jagtap 		no flags Details
OrgA manifest upload (76.66 KB, image/png)
2016-11-02 09:21 UTC, Sanket Jagtap 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 871886 0 unspecified CLOSED Manifest Import Error With Content IDs Changing Upstream 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1392097 0 high CLOSED upgrade fails for candlepin-0.9.54.13-1 if manual patch was applied 2021-02-22 00:41:40 UTC
Red Hat Knowledge Base (Solution) 2726511 0 None None None 2016-10-25 17:36:40 UTC
Red Hat Product Errata RHBA-2016:2699 0 normal SHIPPED_LIVE Satellite 6.2.4 Async Bug Release 2016-11-10 13:12:22 UTC

Internal Links: 871886 1392097

Description sthirugn@redhat.com 2016-10-24 17:54:57 UTC 
Description of problem:
Manifest upload on a second org failed

Version-Release number of selected component (if applicable):
# rpm -qa | grep satellite
satellite-installer-6.3.0-1.el7sat.noarch
satellite-cli-6.3.0-1.0.git.7.fb12bf2.el7sat.noarch
tfm-rubygem-foreman_theme_satellite-1.0.0-1.git.2.94b76fc.el7.noarch
satellite-6.3.0-1.0.git.7.fb12bf2.el7sat.noarch

How reproducible:
Always

Steps to Reproduce:
1. Upload a manifest in org - `Default Organization`
2. Create a second manifest from the same portal account and upload to second org.

Actual results:
Manifest upload failed.

See attached errors from:
* production.log:
2016-10-24 12:50:04  [app] [E] Error during manifest import: {"displayMessage"=>"Failed to import archive", "requestUuid"=>"398feb37-f6bd-4709-998a-630f0e8f4a95"}
2016-10-24 12:50:04  [foreman-tasks/action] [E] Failed to import archive (Katello::Errors::CandlepinError)


* /var/log/candlepin/error.log:
2016-10-24 12:50:04,662 [thread=http-bio-8443-exec-7] [req=398feb37-f6bd-4709-998a-630f0e8f4a95, org=secondorg] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - ERROR: duplicate key value violates unique constraint "cp_content_label_key"
  Detail: Key (label)=(rhel-7-server-containers) already exists.
2016-10-24 12:50:04,693 [thread=http-bio-8443-exec-7] [req=398feb37-f6bd-4709-998a-630f0e8f4a95, org=secondorg] ERROR org.candlepin.sync.Importer - Failed to import archive


Expected results:
Manifest upload works successfully.

Additional info:
Comment 1 sthirugn@redhat.com 2016-10-24 17:55:18 UTC 
Created attachment 1213534 [details]
production.log
Comment 2 sthirugn@redhat.com 2016-10-24 17:55:40 UTC 
Created attachment 1213535 [details]
candlepin error.log
Comment 7 Matt Ruzicka 2016-10-24 18:21:45 UTC 
Would we want to make sure users run a backup before modifying the DB as a "Step 0" or is it reasonably safe?
Comment 8 sthirugn@redhat.com 2016-10-24 18:26:51 UTC 
It appears that workaround mentioned in Comment 6 can cause breaking linkages.  The candlepin team is working on a fix.
Comment 10 Barnaby Court 2016-10-24 19:28:09 UTC 
A safer alternative would be the following:

Step 1:
Remove the uniqueness constraint on the cp_content label column and change it to a regular index.

# sudo su - postgres
# psql
postgres=# \connect candlepin
candlepin=# alter table cp_content drop constraint cp_content_label_key;
candlepin=# create index cp_content_label_key on cp_content (label);

Step 2: upload the second org manifest again

Step 3: refresh the first org manifest just to make sure it works.
Comment 17 Sanket Jagtap 2016-11-02 09:19:06 UTC 
Build : Satellite 6.2.4 Snap 1

rpm -qa | grep satellite
tfm-rubygem-foreman_theme_satellite-0.1.33-1.el7sat.noarch
satellite-cli-6.2.4-1.0.el7sat.noarch
satellite-installer-6.2.0.12-1.el7sat.noarch
satellite-6.2.4-1.0.el7sat.noarch

Verification steps:
1. Upload a manifest in org - `Default Organization`
2. Created a second manifest from the same portal account and upload to second org - orgA.

The 2 manifests for the 2 organizations were uploaded successfully

Please see the screenshots
Comment 18 Sanket Jagtap 2016-11-02 09:20:19 UTC 
Created attachment 1216410 [details]
Default org manifest upload
Comment 19 Sanket Jagtap 2016-11-02 09:21:05 UTC 
Created attachment 1216411 [details]
OrgA manifest upload
Comment 20 Ohad Levy 2016-11-03 14:05:40 UTC 
follow up question: I've applied the workaround, but now the upgrade procedure does not work:
Upgrade Step: migrate_candlepin...
[ INFO 2016-11-03 14:04:29 verbose] Upgrade Step: migrate_candlepin...

########## ERROR ############
Error running command: liquibase --driver=org.postgresql.Driver --classpath=/usr/share/java/postgresql-jdbc.jar:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/ --changeLogFile=db/changelog/changelog-update.xml --url=jdbc:postgresql:candlepin --username=candlepin  --password=UwAUXGvbHF3xAQcDeNsXSLah5bo5DoSb migrate -Dcommunity=False
Status code: 65280
Command output: Liquibase update Failed: Migration failed for change set db/changelog/20161025100925-remove-unique-content-name-constraint.xml::20161025100925-1::awood:
     Reason: liquibase.exception.DatabaseException: Error executing SQL ALTER TABLE public.cp_content DROP CONSTRAINT cp_content_label_key: ERROR: constraint "cp_content_label_key" of relation "cp_content" does not exist

Migrating candlepin database
Traceback (most recent call last):
  File "/usr/share/candlepin/cpdb", line 245, in <module>
    dbsetup.update()
  File "/usr/share/candlepin/cpdb", line 69, in update
    self._run_liquibase("db/changelog/changelog-update.xml")
  File "/usr/share/candlepin/cpdb", line 92, in _run_liquibase
    self.community))
  File "/usr/share/candlepin/cpdb", line 38, in run_command
    error_out(command, status, output)
  File "/usr/share/candlepin/cpdb", line 46, in error_out
    raise Exception("Error running command")
Exception: Error running command

[ERROR 2016-11-03 14:04:38 verbose] 
########## ERROR ############
Error running command: liquibase --driver=org.postgresql.Driver --classpath=/usr/share/java/postgresql-jdbc.jar:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/ --changeLogFile=db/changelog/changelog-update.xml --url=jdbc:postgresql:candlepin --username=candlepin  --password=UwAUXGvbHF3xAQcDeNsXSLah5bo5DoSb migrate -Dcommunity=False
Status code: 65280
Command output: Liquibase update Failed: Migration failed for change set db/changelog/20161025100925-remove-unique-content-name-constraint.xml::20161025100925-1::awood:
     Reason: liquibase.exception.DatabaseException: Error executing SQL ALTER TABLE public.cp_content DROP CONSTRAINT cp_content_label_key: ERROR: constraint "cp_content_label_key" of relation "cp_content" does not exist

Migrating candlepin database
Traceback (most recent call last):
  File "/usr/share/candlepin/cpdb", line 245, in <module>
    dbsetup.update()
  File "/usr/share/candlepin/cpdb", line 69, in update
    self._run_liquibase("db/changelog/changelog-update.xml")
  File "/usr/share/candlepin/cpdb", line 92, in _run_liquibase
    self.community))
  File "/usr/share/candlepin/cpdb", line 38, in run_command
    error_out(command, status, output)
  File "/usr/share/candlepin/cpdb", line 46, in error_out
    raise Exception("Error running command")
Exception: Error running command

Upgrade step migrate_candlepin failed. Check logs for more information.


what is the recommended approach to resolve it?
Comment 22 Barnaby Court 2016-11-04 18:41:49 UTC 
Hi, 

Today the simplest option is to recreate the old constraint as the upgrade is failing because it recognizes the database is in an unknown state. 

If there is a way the installer could check whether the manual patch was taken then there is a manual liquibase task that could be used to mark the offending changeset as having already been run. 

http://www.liquibase.org/documentation/ant/marknextchangesetran_ant_task.html

The manual SQL would be the reverse of the current 
alter table cp_content drop constraint cp_content_label_key;
create unique index cp_content_label_key on cp_content (label);
Comment 33 errata-xmlrpc 2016-11-10 08:14:48 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2699
Note You need to log in before you can comment on or make changes to this bug.