Bug 1684454

Summary: [Satellite 6 + Insights]Remediation playbook execution fails with an error: Failed to create job. Ensure your systems are registered in Foreman.
Product: Red Hat Satellite Reporter: Amar Huchchanavar <ahuchcha>
Component: Ansible - Configuration ManagementAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED DUPLICATE QA Contact: Lukas Pramuk <lpramuk>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.4.2CC: ahuchcha, akarimi, andy.speagle, bbuckingham, bkearney, ktordeur, nshaik, qguo, roarora
Target Milestone: Unspecified   
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-09-23 18:10:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Idebug.logs none

Description Amar Huchchanavar 2019-03-01 10:11:57 UTC
Description of problem:
[Satellite 6 + Insights]Remediation playbook execution fails with an error: Failed to create job. Ensure your systems are registered in Foreman.

Version-Release number of selected component (if applicable):
satellite-6.4.1-1.el7sat.noarch

How reproducible:
Not all customers are facing this issue.

>production.log

2019-02-22T14:13:12 [I|app|9f08e] Processing by Api::V2::JobInvocationsController#create as JSON
2019-02-22T14:13:12 [I|app|9f08e]   Parameters: {"job_invocation"=>{"feature"=>"ansible_run_insights_plan", "host_ids"=>"plan_id=40799", "inputs"=>{"organization_id"=>3, "plan_id"=>40799}}, "apiv"=>"v2"}
.
2019-02-22T14:13:13 [E|bac|9f08e] external method 'search_by_plan_id' failed with error: undefined method `id' for nil:NilClass (ScopedSearch::QueryNotSupported)
/opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.3/lib/scoped_search/query_builder.rb:408:in `rescue in to_ext_method_sql'
/opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.3/lib/scoped_search/query_builder.rb:405:in `to_ext_method_sql'
/opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.3/lib/scoped_search/query_builder.rb:213:in `sql_test'
/opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.3/lib/scoped_search/query_builder.rb:550:in `sql_test'
/opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.3/lib/scoped_search/query_builder.rb:497:in `to_single_field_sql'
/opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.3/lib/scoped_search/query_builder.rb:509:in `to_sql'
/opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.3/lib/scoped_search/query_builder.rb:54:in `build_find_params'
/opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.3/lib/scoped_search/query_builder.rb:22:in `build_query'
/opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.3/lib/scoped_search/definition.rb:300:in `block in register_named_scope!'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/delegation.rb:90:in `public_send'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/delegation.rb:90:in `block in method_missing'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation.rb:336:in `scoping'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/delegation.rb:90:in `method_missing'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.5.6/app/models/targeting.rb:43:in `block in resolve_hosts!'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:94:in `as'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.5.6/app/models/targeting.rb:43:in `resolve_hosts!'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.5.6/app/lib/actions/remote_execution/run_hosts_job.rb:30:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action.rb:493:in `block (3 levels) in execute_plan'

the problem appears to be ,
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.5.6/app/models/targeting.rb:43:in `block in resolve_hosts!'

Comment 10 Amar Huchchanavar 2019-03-21 04:33:25 UTC
Created attachment 1546303 [details]
Idebug.logs

Comment 11 Nagoor Shaik 2019-05-10 02:13:53 UTC
Hello Brad,

Here is the root cause for the reported bug, problematic system was registered with a short name (client) and when this client is registered to insights, it reports its hostname as FQDN. And when running the remediation scripts it does a lookup with FQDN and unable to find the same returns a nil ID

Executing the same code from /opt/theforeman/tfm/root/usr/share/gems/gems/redhat_access-2.2.0/app/models/redhat_access/concerns/host_managed_extensions.rb reveals that empty Host IDs are returned when doing the lookup

// snippet from the foreman-rake console //

irb(main):006:0> hostname_rules_relation.keys
=> ["client.example.com"]                                    -----> this is the Host known to Insights 
irb(main):007:0> host_ids = Host::Managed.where(:name => hostname_rules_relation.keys)
=> #<ActiveRecord::Relation []>                          -----> when tried to lookup for the Host from Foreman DB, it didn't find any records with a FQDN
irb(main):008:0> host_ids = Host::Managed.where(:name => hostname_rules_relation.keys).pluck(:id)
=> []                                                                ----> nil host ID returned

Confirming the host record from Foreman 

# hammer host list --search "name ~ client"
----|----------------|------------------|------------|------------|-------------------|--------------|----------------------
ID  | NAME           | OPERATING SYSTEM | HOST GROUP | IP         | MAC               | CONTENT VIEW | LIFECYCLE ENVIRONMENT
----|----------------|------------------|------------|------------|-------------------|--------------|----------------------
21 | client | RHEL Server 7.1 |            | 192.168.122.12 | aa:bb:cc:dd:00:c1 | RHEL7        | Library
----|----------------|------------------|------------|------------|-------------------|--------------|----------------------

This exactly matches what's reported in https://bugzilla.redhat.com/show_bug.cgi?id=1644832 and it's fixed in Satellite 6.5 Snap 17 release.

I think we can close out this bug in favor of https://bugzilla.redhat.com/show_bug.cgi?id=1644832

Comment 15 Brad Buckingham 2019-07-10 20:52:35 UTC
Hi Nagoor,

Before we close bugzilla as a duplicate of bug 1644832 as suggested comment 11, is there a separate bugzilla tracking the redhataccess PR referenced in comment 12?  If not, we should probably create one and associate that PR to it.

Comment 17 Nagoor Shaik 2019-07-15 10:37:12 UTC
Hello Brad,

As Rohan opened a new bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1729843 for handling upper case hostname with insights. We are good to close this one out as a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1644832

Comment 18 Brad Buckingham 2019-09-23 18:10:17 UTC
Thanks Nagoor!

*** This bug has been marked as a duplicate of bug 1644832 ***