Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
The scenario outlined below still occurs so we are cloning this to a new BZ.
I hit this exact error restoring with rubygem-foreman_maintain-0.6.14-1.el7sat.noarch which our latest shipping version with 6.8.z and a large 6.7 database:
...
--------------------------------------------------------------------------------
Restore foreman postgresql dump from backup:
| Restoring foreman dump [FAIL]
Failed executing runuser - postgres -c 'pg_restore -C -d postgres /backup/foreman.dump', exit status 1:
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 5088; 2606 22600 CONSTRAINT remote_execution_features_pkey foreman
pg_restore: [archiver (db)] could not execute query: ERROR: could not create unique index "remote_execution_features_pkey"
DETAIL: Key (id)=(10) is duplicated.
Command was: ALTER TABLE ONLY remote_execution_features
ADD CONSTRAINT remote_execution_features_pkey PRIMARY KEY (id);
+++ This bug was initially created as a clone of Bug #1847189 +++
Description of problem:
foreman-maintain restore fails on large postgresql databases.
Version-Release number of selected component (if applicable):
Satellite 6.7.0
How reproducible:
Consistently
Steps to Reproduce:
1.Attempt to restore a Satellite with a postgresql database so large, it requires more than 30 minutes to restore.
2.
3.
Actual results:
Restore fails with postgresql errors, and when you go to the satellite Web UI, you receive a Web Appliation error.
Expected results:
Restore is successful and Satellite Web UI functions.
Additional info:
This is due to 2 foreman-rake commands running via cron on the satellite every 30 minutes. If they run before the postgresql restores certain tables in the foreman database, the commands reinitialze the tables with default values. Then when the restore process restores those tables, it creates duplicate entries with duplicate id field values. This prevents the primary key constraint from applying correctly, resulting in postgresql errors. As the tables do not have a primary key, the Web Application fails.
The problematic sql errors are:
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 5163; 2606 21956 CONSTRAINT remote_execution_features_pkey foreman
pg_restore: [archiver (db)] could not execute query: ERROR: could not create unique index "remote_execution_features_pkey"
DETAIL: Key (id)=(7) is duplicated.
Command was: ALTER TABLE ONLY remote_execution_features
ADD CONSTRAINT remote_execution_features_pkey PRIMARY KEY (id);
pg_restore: [archiver (db)] Error from TOC entry 5181; 2606 21976 CONSTRAINT settings_pkey foreman
pg_restore: [archiver (db)] could not execute query: ERROR: could not create unique index "settings_pkey"
DETAIL: Key (id)=(58) is duplicated.
Command was: ALTER TABLE ONLY settings
ADD CONSTRAINT settings_pkey PRIMARY KEY (id);
pg_restore: [archiver (db)] Error from TOC entry 5179; 1259 22325 INDEX index_settings_on_name foreman
pg_restore: [archiver (db)] could not execute query: ERROR: could not create unique index "index_settings_on_name"
DETAIL: Key (name)=(katello_default_provision) is duplicated.
Command was: CREATE UNIQUE INDEX index_settings_on_name ON settings USING btree (name);
pg_restore: [archiver (db)] Error from TOC entry 5438; 2606 22652 FK CONSTRAINT fk_rails_8642ef3b5e foreman
pg_restore: [archiver (db)] could not execute query: ERROR: there is no unique constraint matching given keys for referenced table "remote_execution_features"
Command was: ALTER TABLE ONLY job_invocations
ADD CONSTRAINT fk_rails_8642ef3b5e FOREIGN KEY (remote_execution_feature_id) REFERENCES...
pg_restore: [archiver (db)] Error from TOC entry 6205; 0 0 ACL ansible_roles foreman
Successful Workaround: have foreman-maintain stop the crond.service on Satelltue while the restore is running.
The scenario outlined below still occurs so we are cloning this to a new BZ. I hit this exact error restoring with rubygem-foreman_maintain-0.6.14-1.el7sat.noarch which our latest shipping version with 6.8.z and a large 6.7 database: ... -------------------------------------------------------------------------------- Restore foreman postgresql dump from backup: | Restoring foreman dump [FAIL] Failed executing runuser - postgres -c 'pg_restore -C -d postgres /backup/foreman.dump', exit status 1: pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC entry 5088; 2606 22600 CONSTRAINT remote_execution_features_pkey foreman pg_restore: [archiver (db)] could not execute query: ERROR: could not create unique index "remote_execution_features_pkey" DETAIL: Key (id)=(10) is duplicated. Command was: ALTER TABLE ONLY remote_execution_features ADD CONSTRAINT remote_execution_features_pkey PRIMARY KEY (id); +++ This bug was initially created as a clone of Bug #1847189 +++ Description of problem: foreman-maintain restore fails on large postgresql databases. Version-Release number of selected component (if applicable): Satellite 6.7.0 How reproducible: Consistently Steps to Reproduce: 1.Attempt to restore a Satellite with a postgresql database so large, it requires more than 30 minutes to restore. 2. 3. Actual results: Restore fails with postgresql errors, and when you go to the satellite Web UI, you receive a Web Appliation error. Expected results: Restore is successful and Satellite Web UI functions. Additional info: This is due to 2 foreman-rake commands running via cron on the satellite every 30 minutes. If they run before the postgresql restores certain tables in the foreman database, the commands reinitialze the tables with default values. Then when the restore process restores those tables, it creates duplicate entries with duplicate id field values. This prevents the primary key constraint from applying correctly, resulting in postgresql errors. As the tables do not have a primary key, the Web Application fails. The problematic sql errors are: pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC entry 5163; 2606 21956 CONSTRAINT remote_execution_features_pkey foreman pg_restore: [archiver (db)] could not execute query: ERROR: could not create unique index "remote_execution_features_pkey" DETAIL: Key (id)=(7) is duplicated. Command was: ALTER TABLE ONLY remote_execution_features ADD CONSTRAINT remote_execution_features_pkey PRIMARY KEY (id); pg_restore: [archiver (db)] Error from TOC entry 5181; 2606 21976 CONSTRAINT settings_pkey foreman pg_restore: [archiver (db)] could not execute query: ERROR: could not create unique index "settings_pkey" DETAIL: Key (id)=(58) is duplicated. Command was: ALTER TABLE ONLY settings ADD CONSTRAINT settings_pkey PRIMARY KEY (id); pg_restore: [archiver (db)] Error from TOC entry 5179; 1259 22325 INDEX index_settings_on_name foreman pg_restore: [archiver (db)] could not execute query: ERROR: could not create unique index "index_settings_on_name" DETAIL: Key (name)=(katello_default_provision) is duplicated. Command was: CREATE UNIQUE INDEX index_settings_on_name ON settings USING btree (name); pg_restore: [archiver (db)] Error from TOC entry 5438; 2606 22652 FK CONSTRAINT fk_rails_8642ef3b5e foreman pg_restore: [archiver (db)] could not execute query: ERROR: there is no unique constraint matching given keys for referenced table "remote_execution_features" Command was: ALTER TABLE ONLY job_invocations ADD CONSTRAINT fk_rails_8642ef3b5e FOREIGN KEY (remote_execution_feature_id) REFERENCES... pg_restore: [archiver (db)] Error from TOC entry 6205; 0 0 ACL ansible_roles foreman Successful Workaround: have foreman-maintain stop the crond.service on Satelltue while the restore is running.