Bug 1364175

(In reply to Josh Foots from comment #0)
> Description of problem:
> Upgrade Step: update_subscription_facet_backend_data (this may take a while)
> ...
> 
> This step fails with a traceback 
> 
> Version-Release number of selected component (if applicable):
> 
> 
> How reproducible:
> 
> Very
> 
> Steps to Reproduce:
> 1. Install 6.1.9 with Virt-Who and Hypervisors.
> 2. Upgrade to 6.2.0
> 3.
> 
> Actual results:
> 
> Upgrade step update_subscription_facet_backend_data failed. Check logs for
> more information.
> 
> Expected results:
> 
> Upgrade completes successfully.
> 
> 
> Additional info:
> 
> The current workaround is to stop virt-who. Delete the Hypervisors and then
> do the upgrade. The upgrade will complete successfully and you can start
> virt-who up to register the hypvervisors and do guest to host mapping.

just want to be clear.

This workaround, is not what we need to do. This of course will work but running the step manually while virt-who service is off works just fine so far.
--
The current workaround is to stop virt-who. Delete the Hypervisors and then do the upgrade. The upgrade will complete successfully and you can start virt-who up to register the hypvervisors and do guest to host mapping.
--

not needed ^

FYI, this seems to be the traceback:

[ INFO 2016-08-03 15:25:27 main] Upgrade Step: update_subscription_facet_backend_data (this may take a while) ...
[ERROR 2016-08-03 15:36:30 main] rake aborted!
ActiveRecord::RecordInvalid: Validation failed: Name is invalid
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/validations.rb:57:in `save!'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/attribute_methods/dirty.rb:29:in `save!'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:273:in `block in save!'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:329:in `block in with_transaction_returning_status'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `block in transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in `within_new_transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in `transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:326:in `with_transaction_returning_status'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:273:in `save!'
/usr/share/foreman/app/models/host/base.rb:199:in `set_interfaces'
/usr/share/foreman/app/models/host/base.rb:179:in `populate_fields_from_facts'
/usr/share/foreman/app/models/host/managed.rb:474:in `populate_fields_from_facts'
/usr/share/foreman/app/models/host/base.rb:153:in `import_facts'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/app/models/katello/host/subscription_facet.rb:82:in `update_facts'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake:16:in `block (5 levels) in <top (required)>'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `block (2 levels) in find_each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `block in find_each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:125:in `find_in_batches'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:51:in `find_each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/querying.rb:9:in `find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake:8:in `block (4 levels) in <top (required)>'
Tasks: TOP => katello:upgrades:3.0:update_subscription_facet_backend_data

Has this been able to be reproduced internally?  

Do we have the virt-who configs of the affected virt-who clients?

Would it be possible to get the facts of the hypervisor when this is happening?  You should be able to modify:

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.*/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake

diff --git a/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake b/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake
index 7a08753..9a941c0 100644
--- a/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake
+++ b/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake
@@ -10,6 +10,9 @@ namespace :katello do
             candlepin_attrs = subscription_facet.candlepin_consumer.consumer_attributes
             subscription_facet.import_database_attributes(candlepin_attrs)
 
+            print subscription_facet.host.name
+            print candlepin_attrs.inspect
+            print "\n\n\n"
             subscription_facet.host = ::Host::Managed.find(subscription_facet.host_id)
             subscription_facet.save!
 

and run:

foreman-rake katello:upgrades:3.0:update_subscription_facet_backend_data &> output.txt

(In reply to Justin Sherrill from comment #5)
> Has this been able to be reproduced internally?  
> 
> Do we have the virt-who configs of the affected virt-who clients?
> 
> Would it be possible to get the facts of the hypervisor when this is
> happening?  You should be able to modify:
> 
> /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.*/lib/katello/
> tasks/upgrades/3.0/update_subscription_facet_backend_data.rake
> 
> diff --git
> a/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake
> b/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake
> index 7a08753..9a941c0 100644
> ---
> a/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake
> +++
> b/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake
> @@ -10,6 +10,9 @@ namespace :katello do
>              candlepin_attrs =
> subscription_facet.candlepin_consumer.consumer_attributes
>              subscription_facet.import_database_attributes(candlepin_attrs)
>  
> +            print subscription_facet.host.name
> +            print candlepin_attrs.inspect
> +            print "\n\n\n"
>              subscription_facet.host =
> ::Host::Managed.find(subscription_facet.host_id)
>              subscription_facet.save!
>  
> 
> and run:
> 
> foreman-rake katello:upgrades:3.0:update_subscription_facet_backend_data &>
> output.txt

We just hit this on another upgrade, and the satellite has no virt-who involved just fyi...

Greg & Kathryn,

Engineering would love to assist, but as this bug seems to require very specific customer data we are going to need more information (as per https://bugzilla.redhat.com/show_bug.cgi?id=1364175#c5).

Ideally we would have an internal reproducer, but if not the second best thing is the information from that comment.

Created attachment 1189623 [details]
output for comment5

Thanks stefan!  

It looks like the cause of this is possibly that the user(s) here have virt-who hosts with underscores in their name.  

I've uploaded a new copy of update_subscription_facet_backend_data.rake.  Simply download it and replace this file:

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.*/lib/katello/tasks/upgrades/3.0/update_subscription_facet_backend_data.rake

If you are still seeing issues, please attach the output from that script just like Stefan did.

-Justin

Created attachment 1189635 [details]
new update_subscription_facet_backend_data.rake

Created attachment 1189689 [details]
requested output. ( no virt-who )

(In reply to Justin Sherrill from comment #12)
> Created attachment 1189635 [details]
> new update_subscription_facet_backend_data.rake

Can you look at the attachment from your request on comment 5, would the fix you just uploaded be for my customer that has no hypervisors/virt-who as well?

Thanks.

Kathryn,

It does look like the same issue. Can you try my attached fix?

-Justin

Created attachment 1189789 [details]
Case 01675939 output for comment 5

Created attachment 1189790 [details]
Case 01675939 output for comment 12

Created redmine issue http://projects.theforeman.org/issues/16117 from this bug

Created attachment 1191691 [details]
latest (8/17) update_subscription_facet_backend_data.rake

This updated update_subscription_facet file obsoletes the prior one in this bugzilla

Moving to POST since upstream bug http://projects.theforeman.org/issues/16117 has been closed

Created attachment 1191975 [details]
latest (8/18) update_subscription_facet_backend_data.rake

Verified in Satellite 6.2.1 Snap 1.3. Upgrade from Satellite 6.1.8 executed without issue.

Hypervisors with underscores in their name were changed to dashes. For example, bulg_aria.hq.gss_lab.rdu.redhat.com became virt-who-bulg-aria.hq.gss-lab.rdu.redhat.com-1

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1643