Bug 1333027

Summary: Migration error in foreman_bootdisk upgrading from 6.1 to 6.2
Product: Red Hat Satellite Reporter: Brad Buckingham <bbuckingham>
Component: Audit LogAssignee: Martin Bacovsky <mbacovsk>
Status: CLOSED ERRATA QA Contact: Sachin Ghai <sghai>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2.0CC: cwelton, ehelms, lzap, sghai
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
URL: http://projects.theforeman.org/issues/14965
Whiteboard:
Fixed In Version: rubygem-foreman-bootdisk-6.1.0.1-1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-27 09:31:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1335807    
Attachments:
Description Flags
Installer log during upgrade none

Description Brad Buckingham 2016-05-04 14:25:05 UTC
Description of problem:

Attempting to upgrade to an early version of Satellite 6.2 Beta (internal build) from 6.1 is resulting in an error during the migration of the foreman_bootdisk plugin.

Version-Release number of selected component (if applicable):

foreman-1.11.0.17-1.el7sat.noarch
tfm-rubygem-foreman_bootdisk-6.1.0-1.el7sat.noarch

How reproducible:
Always

Steps to Reproduce:
1. Install Satellite 6.1
2. Perform a Satellite 6.2 upgrade (e.g. update repos/packages, foreman-installer --upgrade)

Actual results:

Partial error: (Full log will be attached)

[ INFO 2016-05-04 12:50:12 main] Upgrade Step: migrate_foreman...
[ERROR 2016-05-04 12:50:20 main] /opt/theforeman/tfm/root/usr/share/gems/gems/docker-api-1.17.0/lib/docker.rb:103: warning: circular argument reference - connection
rake aborted!
NoMethodError: undefined method `request_uuid' for #<Audited::Adapters::ActiveRecord::Audit:0x0000000a4b69a0>
/opt/rh/rh-ror41/root/usr/share/gems/gems/activemodel-4.1.5/lib/active_model/attribute_methods.rb:435:in `method_missing'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/attribute_methods.rb:208:in `method_missing'
...
...
/usr/share/foreman/app/models/setting.rb:201:in `block in create_existing'
/usr/share/foreman/app/models/setting.rb:210:in `bypass_readonly'
/usr/share/foreman/app/models/setting.rb:197:in `create_existing'
/usr/share/foreman/app/models/setting.rb:173:in `create'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_bootdisk-6.1.0/app/models/setting/bootdisk.rb:17:in `block (2 levels) in load_defaults'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_bootdisk-6.1.0/app/models/setting/bootdisk.rb:17:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_bootdisk-6.1.0/app/models/setting/bootdisk.rb:17:in `block in load_defaults'
...
...

Expected results:

No error.

Additional info:

Comment 1 Brad Buckingham 2016-05-04 14:25:34 UTC
Created attachment 1153896 [details]
Installer log during upgrade

Comment 2 Lukas Zapletal 2016-05-04 14:54:53 UTC
I don't believe this has anything to do with bootdisk, it simply comes first (abrt was likely not installed or it does not have any setting). I don't know exactly what is going on in that "Upgrade Step", flipping on Martin who I believe maintain this stuff.

For the record, there were no changes in migrations or settings in this release.

Comment 3 Lukas Zapletal 2016-05-04 15:01:41 UTC
Just for the record, this happens when you update autided gem first and then try to migrate database while there is also a setting change during initial boot. I wonder why we didn't see this during Foreman 1.11 RC upgrade.

http://projects.theforeman.org/issues/10409

I *think* we need to prevent from upgrading audited gem until database is fully migrated. Chicken and egg problem?

Comment 4 Lukas Zapletal 2016-05-04 15:02:21 UTC
Check audited gem before upgrade and after. What were the versions? Perhaps the one in Foreman was newer than in Satellite.

Comment 5 Lukas Zapletal 2016-05-04 15:05:00 UTC
Two possible solutions - either turn audited off for Settings creation from initializers (in all initializers actually) or create special 00_initializer that will do audited table "mini-migration" before any record is touched.

Comment 8 Martin Bacovsky 2016-05-09 11:15:26 UTC
Created redmine issue http://projects.theforeman.org/issues/14965 from this bug

Comment 9 Bryan Kearney 2016-05-09 14:18:57 UTC
Upstream bug component is Audit Log

Comment 10 Bryan Kearney 2016-05-09 16:18:14 UTC
Moving to POST since upstream bug http://projects.theforeman.org/issues/14965 has been closed

Comment 11 Brad Buckingham 2016-05-09 20:27:16 UTC
*** Bug 1333744 has been marked as a duplicate of this bug. ***

Comment 13 Sachin Ghai 2016-06-03 08:05:54 UTC
Verified with upgrade from sat6.1.9 -> sat6.2 GA snap14.2 on rhel7.

Upgrade was successfully completed. I don't see any issue with migration step.

SATELLITE UPGRADE started at: Fri Jun  3 02:18:11 2016
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] run: satellite-installer --scenario satellite --upgrade
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrading...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: stop_services...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: start_databases...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: update_http_conf...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: migrate_pulp...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: fix_pulp_httpd_conf...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: start_httpd...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: start_qpidd...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: start_pulp...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: migrate_candlepin...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: start_tomcat...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: fix_katello_settings_file...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: migrate_foreman...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: remove_nodes_distributors...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: Running installer...

Done                                               [100%] []Installing             Done                                               [100%] []
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out:   The full log is at /var/log/foreman-installer/satellite.log
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: restart_services...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: db_seed...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: import_package_groups (this may take a while) ...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: import_rpms (this may take a while) ...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: import_distributions (this may take a while) ...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: import_puppet_modules (this may take a while) ...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: import_subscriptions (this may take a while) ...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: elasticsearch_message...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Elasticsearch has been removed as a dependency.  The database files can be removed manually with #rm -rf /var/lib/elasticsearch.  Some packages are no longer needed and can be removed:  #rpm -e ruby193-rubygem-tire tfm-rubygem-tire elasticsearch sigar-java sigar snappy-java lucene4-contrib lucene4
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: remove_docker_v1_content (this may take a while) ...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: remove_gutterball...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Katello upgrade completed!
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: 

SATELLITE UPGRADE finished at: Fri Jun  3 03:07:19 2016

Comment 14 Sachin Ghai 2016-06-03 08:31:23 UTC
Just to correct: Verification is done with upgrade from sat6.1.9 -> sat6.2 GA snap14.1 on rhel7. Rpm version is 14.2

satellite-6.2.0-14.2.el7sat.noarch

Comment 16 errata-xmlrpc 2016-07-27 09:31:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1502