Bug 1364175 - update_subscription_facet_backend_data step fails on 6.2 upgrade
Summary: update_subscription_facet_backend_data step fails on 6.2 upgrade
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Upgrades
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
high
high vote
Target Milestone: Unspecified
Assignee: Justin Sherrill
QA Contact: jcallaha
URL: http://projects.theforeman.org/issues...
Whiteboard:
Keywords: Triaged
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-04 15:48 UTC by Josh Foots
Modified: 2019-06-13 21:25 UTC (History)
12 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2016-08-22 06:36:13 UTC


Attachments (Terms of Use)
output for comment5 (1.87 MB, text/plain)
2016-08-10 12:50 UTC, Stefan Nemeth
no flags Details
new update_subscription_facet_backend_data.rake (1.42 KB, text/plain)
2016-08-10 14:12 UTC, Justin Sherrill
no flags Details
requested output. ( no virt-who ) (258.09 KB, text/plain)
2016-08-10 15:28 UTC, Kathryn Dixon
no flags Details
Case 01675939 output for comment 5 (897.24 KB, application/x-gzip)
2016-08-10 19:50 UTC, Rick Dixon
no flags Details
Case 01675939 output for comment 12 (897.24 KB, application/x-gzip)
2016-08-10 19:51 UTC, Rick Dixon
no flags Details
latest (8/17) update_subscription_facet_backend_data.rake (2.58 KB, text/plain)
2016-08-17 17:05 UTC, Mike McCune
no flags Details
latest (8/18) update_subscription_facet_backend_data.rake (2.62 KB, text/plain)
2016-08-18 18:50 UTC, Justin Sherrill
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1643 normal SHIPPED_LIVE Satellite 6.2 Upgrades Bugfixes 2016-08-22 10:35:37 UTC
Foreman Issue Tracker 16117 None None None 2016-08-15 23:06 UTC
Red Hat Knowledge Base (Solution) 2485931 None None None 2016-08-16 00:11 UTC

Comment 3 Kathryn Dixon 2016-08-05 15:14:42 UTC
(In reply to Josh Foots from comment #0)
> Description of problem:
> Upgrade Step: update_subscription_facet_backend_data (this may take a while)
> ...
> 
> This step fails with a traceback 
> 
> Version-Release number of selected component (if applicable):
> 
> 
> How reproducible:
> 
> Very
> 
> Steps to Reproduce:
> 1. Install 6.1.9 with Virt-Who and Hypervisors.
> 2. Upgrade to 6.2.0
> 3.
> 
> Actual results:
> 
> Upgrade step update_subscription_facet_backend_data failed. Check logs for
> more information.
> 
> Expected results:
> 
> Upgrade completes successfully.
> 
> 
> Additional info:
> 
> The current workaround is to stop virt-who. Delete the Hypervisors and then
> do the upgrade. The upgrade will complete successfully and you can start
> virt-who up to register the hypvervisors and do guest to host mapping.

just want to be clear.

This workaround, is not what we need to do. This of course will work but running the step manually while virt-who service is off works just fine so far.
--
The current workaround is to stop virt-who. Delete the Hypervisors and then do the upgrade. The upgrade will complete successfully and you can start virt-who up to register the hypvervisors and do guest to host mapping.
--

not needed ^

Comment 4 Justin Sherrill 2016-08-05 15:15:43 UTC
FYI, this seems to be the traceback:

[ INFO 2016-08-03 15:25:27 main] Upgrade Step: update_subscription_facet_backend_data (this may take a while) ...
[ERROR 2016-08-03 15:36:30 main] rake aborted!
ActiveRecord::RecordInvalid: Validation failed: Name is invalid
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/validations.rb:57:in `save!'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/attribute_methods/dirty.rb:29:in `save!'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:273:in `block in save!'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:329:in `block in with_transaction_returning_status'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `block in transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in `within_new_transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in `transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:326:in `with_transaction_returning_status'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:273:in `save!'
/usr/share/foreman/app/models/host/base.rb:199:in `set_interfaces'
/usr/share/foreman/app/models/host/base.rb:179:in `populate_fields_from_facts'
/usr/share/foreman/app/models/host/managed.rb:474:in `populate_fields_from_facts'
/usr/share/foreman/app/models/host/base.rb:153:in `import_facts'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/app/models/katello/host/subscription_facet.rb:82:in `update_facts'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake:16:in `block (5 levels) in <top (required)>'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `block (2 levels) in find_each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `block in find_each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:125:in `find_in_batches'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:51:in `find_each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/querying.rb:9:in `find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake:8:in `block (4 levels) in <top (required)>'
Tasks: TOP => katello:upgrades:3.0:update_subscription_facet_backend_data

Comment 5 Justin Sherrill 2016-08-05 16:20:52 UTC
Has this been able to be reproduced internally?  

Do we have the virt-who configs of the affected virt-who clients?

Would it be possible to get the facts of the hypervisor when this is happening?  You should be able to modify:

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.*/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake

diff --git a/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake b/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake
index 7a08753..9a941c0 100644
--- a/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake
+++ b/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake
@@ -10,6 +10,9 @@ namespace :katello do
             candlepin_attrs = subscription_facet.candlepin_consumer.consumer_attributes
             subscription_facet.import_database_attributes(candlepin_attrs)
 
+            print subscription_facet.host.name
+            print candlepin_attrs.inspect
+            print "\n\n\n"
             subscription_facet.host = ::Host::Managed.find(subscription_facet.host_id)
             subscription_facet.save!
 

and run:

foreman-rake katello:upgrades:3.0:update_subscription_facet_backend_data &> output.txt

Comment 8 Kathryn Dixon 2016-08-09 23:03:59 UTC
(In reply to Justin Sherrill from comment #5)
> Has this been able to be reproduced internally?  
> 
> Do we have the virt-who configs of the affected virt-who clients?
> 
> Would it be possible to get the facts of the hypervisor when this is
> happening?  You should be able to modify:
> 
> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.*/lib/katello/
> tasks/upgrades/3.0/update_subscription_facet_backend_data.rake
> 
> diff --git
> a/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake
> b/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake
> index 7a08753..9a941c0 100644
> ---
> a/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake
> +++
> b/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake
> @@ -10,6 +10,9 @@ namespace :katello do
>              candlepin_attrs =
> subscription_facet.candlepin_consumer.consumer_attributes
>              subscription_facet.import_database_attributes(candlepin_attrs)
>  
> +            print subscription_facet.host.name
> +            print candlepin_attrs.inspect
> +            print "\n\n\n"
>              subscription_facet.host =
> ::Host::Managed.find(subscription_facet.host_id)
>              subscription_facet.save!
>  
> 
> and run:
> 
> foreman-rake katello:upgrades:3.0:update_subscription_facet_backend_data &>
> output.txt

We just hit this on another upgrade, and the satellite has no virt-who involved just fyi...

Comment 9 Justin Sherrill 2016-08-10 12:22:50 UTC
Greg & Kathryn,

Engineering would love to assist, but as this bug seems to require very specific customer data we are going to need more information (as per https://bugzilla.redhat.com/show_bug.cgi?id=1364175#c5).

Ideally we would have an internal reproducer, but if not the second best thing is the information from that comment.

Comment 10 Stefan Nemeth 2016-08-10 12:50 UTC
Created attachment 1189623 [details]
output for comment5

Comment 11 Justin Sherrill 2016-08-10 14:11:34 UTC
Thanks stefan!  

It looks like the cause of this is possibly that the user(s) here have virt-who hosts with underscores in their name.  

I've uploaded a new copy of update_subscription_facet_backend_data.rake.  Simply download it and replace this file:

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.*/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake

If you are still seeing issues, please attach the output from that script just like Stefan did.

-Justin

Comment 12 Justin Sherrill 2016-08-10 14:12 UTC
Created attachment 1189635 [details]
new update_subscription_facet_backend_data.rake

Comment 13 Kathryn Dixon 2016-08-10 15:28 UTC
Created attachment 1189689 [details]
requested output. ( no virt-who )

Comment 14 Kathryn Dixon 2016-08-10 16:02:29 UTC
(In reply to Justin Sherrill from comment #12)
> Created attachment 1189635 [details]
> new update_subscription_facet_backend_data.rake

Can you look at the attachment from your request on comment 5, would the fix you just uploaded be for my customer that has no hypervisors/virt-who as well?

Thanks.

Comment 15 Justin Sherrill 2016-08-10 16:16:29 UTC
Kathryn,

It does look like the same issue. Can you try my attached fix?

-Justin

Comment 16 Rick Dixon 2016-08-10 19:50 UTC
Created attachment 1189789 [details]
Case 01675939 output for comment 5

Comment 17 Rick Dixon 2016-08-10 19:51 UTC
Created attachment 1189790 [details]
Case 01675939 output for comment 12

Comment 18 Justin Sherrill 2016-08-15 22:35:30 UTC
Created redmine issue http://projects.theforeman.org/issues/16117 from this bug

Comment 19 Mike McCune 2016-08-17 17:05 UTC
Created attachment 1191691 [details]
latest (8/17) update_subscription_facet_backend_data.rake

This updated update_subscription_facet file obsoletes the prior one in this bugzilla

Comment 20 Bryan Kearney 2016-08-18 02:16:56 UTC
Moving to POST since upstream bug http://projects.theforeman.org/issues/16117 has been closed

Comment 21 Justin Sherrill 2016-08-18 18:50 UTC
Created attachment 1191975 [details]
latest (8/18) update_subscription_facet_backend_data.rake

Comment 23 jcallaha 2016-08-19 19:26:03 UTC
Verified in Satellite 6.2.1 Snap 1.3. Upgrade from Satellite 6.1.8 executed without issue.

Hypervisors with underscores in their name were changed to dashes. For example, bulg_aria.hq.gss_lab.rdu.redhat.com became virt-who-bulg-aria.hq.gss-lab.rdu.redhat.com-1

Comment 25 errata-xmlrpc 2016-08-22 06:36:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1643


Note You need to log in before you can comment on or make changes to this bug.