Bug 1961770

I have performed the update to WiredTiger on 2 Satellite 6.9.1 systems.  

The method I used was to fix the chmod typo, and add "true, true" to the execute function paramaters.

Results:

My Lab satellite  (no capsule):

Everything seemed successful, but when I attempted to sync products, I got 404 Rest Client not found.  I had to manually restore the entire mongodb from backup, and then everything worked again.

My customer:

The import of units_rpm.bson failed on both the satellite and capsule.  I attempted to restore just units_rpm.bson.  I ran a mongodb database repair.  All client yum calls for repo metadta worked, but all rpm downloads failed with 404 not found.  I had to manually restore the entire mongodb from backup, and then everything worked again.  

I believe the process to update to wired tiger is somehow inherently flawed.  Something happens to the mongo data as it is being imported by the commands in 31-mongo_storage_engine.rb, making the data unusable by satellite/pulp.  

I believe this can be tested by the following:

Take a Satellite 6.9.1
backup the mongoDB
wipe the current mongoDB and reset mongo to use mmapv1
Restore mongoDB
run the update to WiredTiger
Attempt to sync Products/Repositories
I am confident you will encounter the 404 Rest Client Error.

With the changes in the foreman-installer version that shipped with Satellite 6.9, `execute!` is intended to be used for shell command execution in installer hooks where success is expected and otherwise the installer should fail with an error; `execute` is intended for commands where the possibility of failure is expected and handled in the code.

So I think by changing each instance of `execute(${command})` to `execute(#{command}, true, true)`, there is still a possibility that one of these commands could fail rather silently when instead installer execution should have stopped and reported an error at that point.

I believe the correct patch should be more like:


```
diff --git a/katello/hooks/pre/31-mongo_storage_engine.rb b/katello/hooks/pre/31-mongo_storage_engine.rb
index e7c18e0..1e63428 100644
--- a/katello/hooks/pre/31-mongo_storage_engine.rb
+++ b/katello/hooks/pre/31-mongo_storage_engine.rb
@@ -13,34 +13,33 @@ def migration
   stop_services
   start_services(['rh-mongodb34-mongod'])
   FileUtils.mkdir(export_dir) unless File.directory?(export_dir)
-  File.chmod(0o700, export_dir)
+  File.chmod(0700, export_dir)
   logger.info "Starting mongodump to #{export_dir}"
-  execute("mongodump --host localhost --out #{export_dir}")
+  execute!("mongodump --host localhost --out #{export_dir}")
 
   # Remove datafiles out of MongoDB directory so it will start
   logger.info 'Export done, stopping MongoDB to remove old datafiles'
   stop_services(['rh-mongodb34-mongod'])
   logger.info "Removing contents from #{mongo_dir}"
-  execute("rm -rf #{mongo_dir}/*")
+  execute!("rm -rf #{mongo_dir}/*")
 
   # Import the dump, fail and notify user of backup if restore does not work.
   logger.info 'Changing config to WiredTiger and starting restore.'
-  execute("sed -i.bak -e 's/mmapv1/wiredTiger/g' #{mongo_conf}")
-  execute("mv #{mongo_conf}.bak #{export_dir}")
+  execute!("sed -i.bak -e 's/mmapv1/wiredTiger/g' #{mongo_conf}")
+  execute!("mv #{mongo_conf}.bak #{export_dir}")
   start_services(['rh-mongodb34-mongod'])
   pulp_db = katello ? param('katello', 'pulp_db_name').value : 'pulp_database'
-  execute("mongorestore --host localhost --db=#{pulp_db} --drop --dir=#{export_dir}/#{pulp_db}")
-  unless $CHILD_STATUS.success?
+  unless execute("mongorestore --host localhost --db=#{pulp_db} --drop --dir=#{export_dir}/#{pulp_db}", true, true)
     logger.error 'The restore could not be completed correctly, reverting actions.'
     logger.info 'Stopping MongoDB'
     stop_services(['rh-mongodb34-mongod'])
 
     logger.info 'Restoring old config'
-    execute("mv -f #{export_dir}/mongod.conf.bak #{mongo_conf}")
+    execute!("mv -f #{export_dir}/mongod.conf.bak #{mongo_conf}")
     logger.info "Removing contents in #{mongo_dir}"
-    execute("rm -rf #{mongo_dir}/*")
+    execute!("rm -rf #{mongo_dir}/*")
     logger.info "Restoring database under MMAPV1 storage engine"
-    execute("mongorestore --host localhost --db=#{pulp_db} --drop --dir=#{export_dir}/#{pulp_db}")
+    execute!("mongorestore --host localhost --db=#{pulp_db} --drop --dir=#{export_dir}/#{pulp_db}")
     logger.info 'Starting MongoDB with old config and database files'
     start_services(['rh-mongodb34-mongod'])
     logger.error "Mongo started up in MMAPV1 mode, backup at #{export_dir}"
@@ -49,12 +48,12 @@ def migration
 
   # Remove old data files
   logger.info "Import done, removing old data in #{export_dir}"
-  execute("rm -rf #{export_dir}")
+  execute!("rm -rf #{export_dir}")
 
   # Update Hiera to wiredTiger for installer run
   logger.info 'Changing custom Hiera to use wiredTiger for installer Puppet run.'
-  execute("sed -i -e 's/Added by foreman-installer during upgrade, run the installer with --upgrade-mongo-storage.* to upgrade to WiredTiger./Do not remove/g' #{hiera_file}")
-  execute("sed -i -e 's/mmapv1/wiredTiger/g' #{hiera_file}")
+  execute!("sed -i -e 's/Added by foreman-installer during upgrade, run the installer with --upgrade-mongo-storage.* to upgrade to WiredTiger./Do not remove/g' #{hiera_file}")
+  execute!("sed -i -e 's/mmapv1/wiredTiger/g' #{hiera_file}")
 end
 
 if app_value(:upgrade_mongo_storage_engine)
```


The core logic of the storage engine upgrade was working on previous Satellite versions and should still work with the above patch. If it is indeed broken beyond what is described in this bug, I think a separate bug should be filed.

Are logs available from your customer for the storage engine upgrade that resulted in those 404 responses? I'm not sure whether that would be useful due to the use of `execute` instead of `execute!` for steps where success is both expected and required for the upgrade, but I would be curious to look and see if anything can be gleaned.

Created attachment 1787638 [details]
satellite-installer log from wired tiger upgrade

This is the log from the Wired Tiger upgrade that failed.  After this upgrade I attempted just doing a restore from backup of pulp_database.units_rpm, and later found that pulp would not serve packages.  I then did a full dump and reload of the database.  Then packages were served again. 

The same issue happened on both the Satellite and capsule.

Thanks Jason!

Checking the log, we can see there was an error during the `mongorestore` step, but the hook continued as if that step had been successful:

```
$ grep -C10 'insertion error: EOF' satellite.20210520-091025.log
2021-05-18 11:53:30 [DEBUG ] [root] 2021-05-18T11:53:24.860-0500	finished restoring pulp_database.consumer_groups (0 documents)
2021-05-18 11:53:30 [DEBUG ] [root] 2021-05-18T11:53:24.862-0500	restoring indexes for collection pulp_database.content_catalog from metadata
2021-05-18 11:53:30 [DEBUG ] [root] 2021-05-18T11:53:24.863-0500	finished restoring pulp_database.content_catalog (0 documents)
2021-05-18 11:53:30 [DEBUG ] [root] 2021-05-18T11:53:26.667-0500	[#.......................]           pulp_database.units_rpm  1.17GB/17.5GB   (6.7%)
2021-05-18 11:53:30 [DEBUG ] [root] 2021-05-18T11:53:26.667-0500	[#########...............]  pulp_database.repo_content_units    182MB/466MB  (39.0%)
2021-05-18 11:53:30 [DEBUG ] [root] 2021-05-18T11:53:26.667-0500	
2021-05-18 11:53:30 [DEBUG ] [root] 2021-05-18T11:53:29.667-0500	[#.......................]           pulp_database.units_rpm  1.17GB/17.5GB   (6.7%)
2021-05-18 11:53:30 [DEBUG ] [root] 2021-05-18T11:53:29.667-0500	[#########...............]  pulp_database.repo_content_units    182MB/466MB  (39.0%)
2021-05-18 11:53:30 [DEBUG ] [root] 2021-05-18T11:53:29.667-0500	
2021-05-18 11:53:30 [DEBUG ] [root] 2021-05-18T11:53:30.633-0500	[#.......................]  pulp_database.units_rpm  1.17GB/17.5GB  (6.7%)
2021-05-18 11:53:30 [DEBUG ] [root] 2021-05-18T11:53:30.634-0500	Failed: pulp_database.units_rpm: error restoring from /var/tmp/mongodb_engine_upgrade/pulp_database/units_rpm.bson: insertion error: EOF                                <----- ERROR is here
2021-05-18 11:53:30 [INFO  ] [pre] Import done, removing old data in /var/tmp/mongodb_engine_upgrade
2021-05-18 11:53:30 [DEBUG ] [root] Executing: rm -rf /var/tmp/mongodb_engine_upgrade
2021-05-18 11:53:32 [INFO  ] [pre] Changing custom Hiera to use wiredTiger for installer Puppet run.
2021-05-18 11:53:32 [DEBUG ] [root] Executing: sed -i -e 's/Added by foreman-installer during upgrade, run the installer with --upgrade-mongo-storage.* to upgrade to WiredTiger./Do not remove/g' /etc/foreman-installer/custom-hiera.yaml
2021-05-18 11:53:32 [DEBUG ] [root] Executing: sed -i -e 's/mmapv1/wiredTiger/g' /etc/foreman-installer/custom-hiera.yaml
2021-05-18 11:53:32 [DEBUG ] [pre] Hook /usr/share/foreman-installer/katello/hooks/pre/31-mongo_storage_engine.rb returned true
2021-05-18 11:53:32 [DEBUG ] [pre] Hook /usr/share/foreman-installer/hooks/pre/31-puppet_agent_oauth.rb returned nil
2021-05-18 11:53:32 [DEBUG ] [pre] Hook /usr/share/foreman-installer/hooks/pre/32-install_selinux_packages.rb returned nil
2021-05-18 11:53:32 [DEBUG ] [pre] Hook /usr/share/foreman-installer/hooks/pre/33-pulpcore_assets_permissions.rb returned nil
2021-05-18 11:53:32 [DEBUG ] [pre] Hook /usr/share/foreman-installer/hooks/pre/34-pulpcore_directory_layout.rb returned nil
```

@ehelms is this consistent with the issue you saw in your own testing? should we create a BZ?

Note that my attached MR also includes the change:

```
-  execute("mongorestore --host localhost --db=#{pulp_db} --drop --dir=#{export_dir}/#{pulp_db}")
-  unless $CHILD_STATUS.success?
+  unless execute("mongorestore --host localhost --db=#{pulp_db} --drop --dir=#{export_dir}/#{pulp_db}", true, true)
```

I believe this would at least cause the hook to correctly catch the failure and attempt to revert to the prior configuration. I am unsure at this time what was the exact cause of the underlying mongorestore error: `Failed: pulp_database.units_rpm: error restoring from /var/tmp/mongodb_engine_upgrade/pulp_database/units_rpm.bson: insertion error: EOF`

Created attachment 1787640 [details]
failure from my test lab satellite

I encountered an error rebuilding indexes.  while different errors, seems the same overall situation as the previous customer log I uploaded.

With the release of Satellite 6.11 on the horizon, and with Satellite 6.10 bringing on the transition to Pulp 3 which drops MongoDB we do not intend to do any further fixes or improvements to the MongoDB wiredTiger path. This functionality was a nice to have for users of Pulp 2 and MongoDB but is not needed as part of the upgrade path to Satellite 6.10+. I am therefore opting to CLOSE:WONTFIX this request.

I do not expect us to remove the option from the installer for older releases, and as such it will remain as an option that is broken. A docs update to remove this from the guides will be investigated to reduce confusion.