Bug 1664641 - cannot restore backup what was created before upgrade-mongo-storage-engine
Summary: cannot restore backup what was created before upgrade-mongo-storage-engine
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Backup & Restore
Version: 6.5.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: 6.5.0
Assignee: Martin Bacovsky
QA Contact: Lucie Vrtelova
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-01-09 11:18 UTC by Ales Dujicek
Modified: 2019-11-05 22:43 UTC (History)
6 users (show)

Fixed In Version: rubygem-foreman_maintain-0.3.1-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-14 12:39:45 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 25939 0 Normal Closed cannot restore backup what was created before upgrade-mongo-storage-engine 2020-08-27 14:39:08 UTC
Red Hat Product Errata RHSA-2019:1222 0 None None None 2019-05-14 12:39:54 UTC

Description Ales Dujicek 2019-01-09 11:18:35 UTC
Description of problem:

it fails to restore backup what had been created before mongo storage engine was upgraded to wiredtiger

# foreman-maintain restore /tmp/backup-6.5/satellite-backup-2019-01-08-15-19-56/
Running Restore backup
...
--------------------------------------------------------------------------------
Restore configs from backup:
\ Restoring configs                                                   [OK]  
...
Run installer reset:
\ Installer reset                                                     [FAIL]   
Failed executing yes | satellite-installer -v --reset --disable-system-checks , exit status 6:
[ INFO 2019-01-08T16:42:08 verbose] Dropping Pulp database!
rm -f /var/lib/pulp/init.flag finished successfully!
systemctl stop httpd pulp_workers finished successfully!
Job for rh-mongodb34-mongod.service failed because the control process exited with error code. See "systemctl status rh-mongodb34-mongod.service" and "journalctl -xe" for details.
systemctl start rh-mongodb34-mongod failed! Check the output for error!
[ERROR 2019-01-08T16:42:08 verbose] systemctl start rh-mongodb34-mongod failed! Check the output for error!
MongoDB shell version v3.4.9
connecting to: mongodb://127.0.0.1:27017/pulp_database
2019-01-08T16:42:08.847+0100 W NETWORK  [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2019-01-08T16:42:08.939+0100 E QUERY    [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:237:13
@(connect):1:6
exception: connect failed
mongo pulp_database --eval 'db.dropDatabase();' failed! Check the output for error!
[ERROR 2019-01-08T16:42:08 verbose] mongo pulp_database --eval 'db.dropDatabase();' failed! Check the output for error!
...
[ERROR 2019-01-08T16:42:33 verbose] Jan 08 16:42:08 vm-198-147.lab.eng.pek2.redhat.com mongod.27017[11541]: [initandlisten] exception in initAndListen: 28662 Cannot start server. Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, but the specified storage engine was 'mmapv1'., terminating
...
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/satellite.log
[ INFO 2019-01-08T16:54:31 verbose] All hooks in group post finished
[ INFO 2019-01-08T16:54:31 verbose] Installer finished in 804.191110789 seconds
--------------------------------------------------------------------------------
Scenario [Restore backup] failed.
...

problem is that /etc/* files are restored first (Restore configs from backup) where /etc/opt/rh/rh-mongodb34/mongod.conf contains
storage.engine: mmapv1
so later when it does sth. with mongo, it fails to start because it detects wiredtiger files in /var/lib/mongodb

Version-Release number of selected component (if applicable):
sat 6.5 snap 10
rubygem-foreman_maintain-0.3.0-1.el7sat.noarch

How reproducible:
always

Steps to Reproduce:
1. install satellite 6.3 (last version with old mongo)
2. upgrade to 6.4 
3. upgrade to 6.5
4. foreman-maintain backup ...
5. satellite-installer --upgrade-mongo-storage-engine
6. foreman-maintain restore ...

Comment 3 Brad Buckingham 2019-01-10 16:35:46 UTC
At a minimum, there needs to be some documentation or messaging so that the user knows that they need to do a new backup after the upgrade to mongo-storage-engine.

Comment 4 Craig Donnelly 2019-01-10 17:44:35 UTC
To clarify, is the issue here that the restore reconfigures mongo to start with the mmapv1 storage engine?

Comment 5 Ales Dujicek 2019-01-11 12:30:21 UTC
yes, and it has to do so (at least in case of offline backup) because restored data would be mmpav1

Comment 6 Martin Bacovsky 2019-01-30 08:21:06 UTC
Created redmine issue https://projects.theforeman.org/issues/25939 from this bug

Comment 7 Bryan Kearney 2019-01-31 13:08:39 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/25939 has been resolved.

Comment 11 errata-xmlrpc 2019-05-14 12:39:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222


Note You need to log in before you can comment on or make changes to this bug.