Bug 1847189
| Summary: | foreman-maintain restore fails on large postgresql databases | |||
|---|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Jason Dickerson <jdickers> | |
| Component: | Satellite Maintain | Assignee: | Suraj Patil <supatil> | |
| Status: | CLOSED ERRATA | QA Contact: | Lucie Vrtelova <lvrtelov> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 6.7.0 | CC: | apatel, ehelms, jjeffers, kgaikwad, mmccune, pcreech, pmoravec, supatil | |
| Target Milestone: | 6.8.0 | Keywords: | Triaged | |
| Target Release: | Unused | |||
| Hardware: | x86_64 | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1923486 (view as bug list) | Environment: | ||
| Last Closed: | 2020-10-27 12:38:24 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1923486 | |||
Created redmine issue https://projects.theforeman.org/issues/30164 from this bug Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/30164 has been resolved. Can we get a release of foreman_maintain 0.5.z with this fix in it? Moving this back to 6.8.0 (and dropping from 6.7.4) after discussion with build team and QE. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Satellite 6.8 Satellite Maintenance Release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4365 I hit this exact error restoring with rubygem-foreman_maintain-0.6.14-1.el7sat.noarch which our latest shipping version with 6.8.z and a large 6.7 database:
...
--------------------------------------------------------------------------------
Restore foreman postgresql dump from backup:
| Restoring foreman dump [FAIL]
Failed executing runuser - postgres -c 'pg_restore -C -d postgres /backup/foreman.dump', exit status 1:
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 5088; 2606 22600 CONSTRAINT remote_execution_features_pkey foreman
pg_restore: [archiver (db)] could not execute query: ERROR: could not create unique index "remote_execution_features_pkey"
DETAIL: Key (id)=(10) is duplicated.
Command was: ALTER TABLE ONLY remote_execution_features
ADD CONSTRAINT remote_execution_features_pkey PRIMARY KEY (id);
pg_restore: [archiver (db)] Error from TOC entry 4718; 2606 18751 CONSTRAINT settings_pkey foreman
pg_restore: [archiver (db)] could not execute query: ERROR: could not create unique index "settings_pkey"
DETAIL: Key (id)=(175) is duplicated.
Command was: ALTER TABLE ONLY settings
ADD CONSTRAINT settings_pkey PRIMARY KEY (id);
we may not be stopping crond properly and will clone this bz if so
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days |
Description of problem: foreman-maintain restore fails on large postgresql databases. Version-Release number of selected component (if applicable): Satellite 6.7.0 How reproducible: Consistently Steps to Reproduce: 1.Attempt to restore a Satellite with a postgresql database so large, it requires more than 30 minutes to restore. 2. 3. Actual results: Restore fails with postgresql errors, and when you go to the satellite Web UI, you receive a Web Appliation error. Expected results: Restore is successful and Satellite Web UI functions. Additional info: This is due to 2 foreman-rake commands running via cron on the satellite every 30 minutes. If they run before the postgresql restores certain tables in the foreman database, the commands reinitialze the tables with default values. Then when the restore process restores those tables, it creates duplicate entries with duplicate id field values. This prevents the primary key constraint from applying correctly, resulting in postgresql errors. As the tables do not have a primary key, the Web Application fails. The problematic sql errors are: pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC entry 5163; 2606 21956 CONSTRAINT remote_execution_features_pkey foreman pg_restore: [archiver (db)] could not execute query: ERROR: could not create unique index "remote_execution_features_pkey" DETAIL: Key (id)=(7) is duplicated. Command was: ALTER TABLE ONLY remote_execution_features ADD CONSTRAINT remote_execution_features_pkey PRIMARY KEY (id); pg_restore: [archiver (db)] Error from TOC entry 5181; 2606 21976 CONSTRAINT settings_pkey foreman pg_restore: [archiver (db)] could not execute query: ERROR: could not create unique index "settings_pkey" DETAIL: Key (id)=(58) is duplicated. Command was: ALTER TABLE ONLY settings ADD CONSTRAINT settings_pkey PRIMARY KEY (id); pg_restore: [archiver (db)] Error from TOC entry 5179; 1259 22325 INDEX index_settings_on_name foreman pg_restore: [archiver (db)] could not execute query: ERROR: could not create unique index "index_settings_on_name" DETAIL: Key (name)=(katello_default_provision) is duplicated. Command was: CREATE UNIQUE INDEX index_settings_on_name ON settings USING btree (name); pg_restore: [archiver (db)] Error from TOC entry 5438; 2606 22652 FK CONSTRAINT fk_rails_8642ef3b5e foreman pg_restore: [archiver (db)] could not execute query: ERROR: there is no unique constraint matching given keys for referenced table "remote_execution_features" Command was: ALTER TABLE ONLY job_invocations ADD CONSTRAINT fk_rails_8642ef3b5e FOREIGN KEY (remote_execution_feature_id) REFERENCES... pg_restore: [archiver (db)] Error from TOC entry 6205; 0 0 ACL ansible_roles foreman Successful Workaround: have foreman-maintain stop the crond.service on Satelltue while the restore is running.