Created attachment 1137324 [details]
log for the db:migrate command

We have asked the customer to run the runner script and waiting for input.

runner script execution failed on customer setup. Please let us know 
if there is any problem in the why it's executed. 

[root@wat-cf4-db vmdb]# bin/rails runner /root/runner_script_1315089.rb
[DEPRECATION] `configure` is removed in secure_headers 3.x. Instead use `default`.
/opt/rh/cfme-gemset/gems/railties-4.2.5.1/lib/rails/commands/runner.rb:62:in `eval': /opt/rh/cfme-gemset/gems/railties-4.2.5.1/lib/rails/commands/runner.rb:62: unknown regexp options - rr (SyntaxError)
/opt/rh/cfme-gemset/gems/railties-4.2.5.1/lib/rails/commands/runner.rb:62: syntax error, unexpected tIDENTIFIER, expecting end-of-input
/root/runner_script_1315089.rb
                           ^
        from /opt/rh/cfme-gemset/gems/railties-4.2.5.1/lib/rails/commands/runner.rb:62:in `<top (required)>'
        from /opt/rh/cfme-gemset/gems/railties-4.2.5.1/lib/rails/commands/commands_tasks.rb:123:in `require'
        from /opt/rh/cfme-gemset/gems/railties-4.2.5.1/lib/rails/commands/commands_tasks.rb:123:in `require_command!'
        from /opt/rh/cfme-gemset/gems/railties-4.2.5.1/lib/rails/commands/commands_tasks.rb:90:in `runner'
        from /opt/rh/cfme-gemset/gems/railties-4.2.5.1/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
        from /opt/rh/cfme-gemset/gems/railties-4.2.5.1/lib/rails/commands.rb:17:in `<top (required)>'
        from bin/rails:4:in `require'
        from bin/rails:4:in `<main>'

This error indicates the file, /root/runner_script_1315089.rb, doesn't exist.  Rails 4's error is not very obvious but I recreated it locally by trying to use a non-existing file.  Thankfully, with Rails 5, this error is more obvious:


$ bin/rails runner /root/runner_script_1315089.rb
** Using session_store: ActionDispatch::Session::MemCacheStore
Please specify a valid ruby command or the path of a script to run.
Run 'bin/rails runner -h' for help.


Josh, please verify this works for you by copying the script to the appliance and passing the correct fully qualified path to bin/rails runner.  

What is the plan?  This script will print the invalid YAML data which will have to be checked to see what is wrong and then manually fixed.  It's not exactly straightforward as you need to understand YAML syntax rules to correct any problems and even then, you need to manually edit the invalid data in the database.  Is this the intention?  You probably want to alter the script if you only want to see some of the metadata about the reports to help decide if any of them need to be saved vs. discarded.

Let me know how I can help.

Thanks,
Joe

Created attachment 1142640 [details]
runner script output

Created attachment 1152416 [details]
Script to delete invalid yaml report results.

For each region:

Copy this attachment to vmdb, (/var/www/miq/vmdb) and run it like this:

1) bundle exec rails r clean_invalid_yaml.rb

If there any invalid yaml rows, it will report "Unable to convert..." messages and ask you to run it again like below to delete these rows:

2) DELETE_INVALID_YAML=true bundle exec rails r clean_invalid_yaml.rb

If you run 2) again, it should NOT report any "Unable to convert..." messages.

Colin/Jared, I have run these steps with the provided database but can you confirm my steps.  Thanks!  Joe

Colin, they should run this on each region database where they're hitting this issue.  I would still run the dry-run way first (without DELETE_INVALID_YAML=true) to ensure there's a fairly small number of results before running it the second way.

After this is done everywhere, they should be able to continue to run the migrations they were originally trying to run as reported in this issue.

All of this should be tried in a simulated test environment using production databases to make sure it does what they want.  I'm not quite sure what step in the upgrade/migrate process they were currently at when they hit the issue.