Bug 1315089 - db:migrate failure (<unknown>): could not find expected ':' while scanning a simple key
db:migrate failure (<unknown>): could not find expected ':' while scanning a...
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Appliance (Show other bugs)
All All
high Severity urgent
: GA
: 5.6.0
Assigned To: Joe Rafaniello
Jan Krocil
Depends On:
  Show dependency treegraph
Reported: 2016-03-06 09:17 EST by Josh Carter
Modified: 2016-05-31 09:53 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2016-05-31 09:52:22 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
log for the db:migrate command (4.66 MB, text/plain)
2016-03-17 03:43 EDT, Prasad Mukhedkar
no flags Details
runner script output (70.83 KB, text/plain)
2016-04-01 12:27 EDT, Prasad Mukhedkar
no flags Details
Script to delete invalid yaml report results. (2.20 KB, text/plain)
2016-04-29 17:25 EDT, Joe Rafaniello
no flags Details

  None (edit)
Comment 14 Prasad Mukhedkar 2016-03-17 03:43 EDT
Created attachment 1137324 [details]
log for the db:migrate command
Comment 15 Prasad Mukhedkar 2016-03-17 03:51:52 EDT
We have asked the customer to run the runner script and waiting for input.
Comment 16 Prasad Mukhedkar 2016-03-30 07:54:36 EDT
runner script execution failed on customer setup. Please let us know 
if there is any problem in the why it's executed. 

[root@wat-cf4-db vmdb]# bin/rails runner /root/runner_script_1315089.rb
[DEPRECATION] `configure` is removed in secure_headers 3.x. Instead use `default`.
/opt/rh/cfme-gemset/gems/railties- `eval': /opt/rh/cfme-gemset/gems/railties- unknown regexp options - rr (SyntaxError)
/opt/rh/cfme-gemset/gems/railties- syntax error, unexpected tIDENTIFIER, expecting end-of-input
        from /opt/rh/cfme-gemset/gems/railties- `<top (required)>'
        from /opt/rh/cfme-gemset/gems/railties- `require'
        from /opt/rh/cfme-gemset/gems/railties- `require_command!'
        from /opt/rh/cfme-gemset/gems/railties- `runner'
        from /opt/rh/cfme-gemset/gems/railties- `run_command!'
        from /opt/rh/cfme-gemset/gems/railties- `<top (required)>'
        from bin/rails:4:in `require'
        from bin/rails:4:in `<main>'
Comment 17 Joe Rafaniello 2016-03-30 08:33:07 EDT
This error indicates the file, /root/runner_script_1315089.rb, doesn't exist.  Rails 4's error is not very obvious but I recreated it locally by trying to use a non-existing file.  Thankfully, with Rails 5, this error is more obvious:

$ bin/rails runner /root/runner_script_1315089.rb
** Using session_store: ActionDispatch::Session::MemCacheStore
Please specify a valid ruby command or the path of a script to run.
Run 'bin/rails runner -h' for help.

Josh, please verify this works for you by copying the script to the appliance and passing the correct fully qualified path to bin/rails runner.  

What is the plan?  This script will print the invalid YAML data which will have to be checked to see what is wrong and then manually fixed.  It's not exactly straightforward as you need to understand YAML syntax rules to correct any problems and even then, you need to manually edit the invalid data in the database.  Is this the intention?  You probably want to alter the script if you only want to see some of the metadata about the reports to help decide if any of them need to be saved vs. discarded.

Let me know how I can help.

Comment 18 Prasad Mukhedkar 2016-04-01 12:27 EDT
Created attachment 1142640 [details]
runner script output
Comment 28 Joe Rafaniello 2016-04-29 17:25 EDT
Created attachment 1152416 [details]
Script to delete invalid yaml report results.

For each region:

Copy this attachment to vmdb, (/var/www/miq/vmdb) and run it like this:

1) bundle exec rails r clean_invalid_yaml.rb

If there any invalid yaml rows, it will report "Unable to convert..." messages and ask you to run it again like below to delete these rows:

2) DELETE_INVALID_YAML=true bundle exec rails r clean_invalid_yaml.rb

If you run 2) again, it should NOT report any "Unable to convert..." messages.

Colin/Jared, I have run these steps with the provided database but can you confirm my steps.  Thanks!  Joe
Comment 35 Joe Rafaniello 2016-05-20 09:26:42 EDT
Colin, they should run this on each region database where they're hitting this issue.  I would still run the dry-run way first (without DELETE_INVALID_YAML=true) to ensure there's a fairly small number of results before running it the second way.

After this is done everywhere, they should be able to continue to run the migrations they were originally trying to run as reported in this issue.

All of this should be tried in a simulated test environment using production databases to make sure it does what they want.  I'm not quite sure what step in the upgrade/migrate process they were currently at when they hit the issue.

Note You need to log in before you can comment on or make changes to this bug.