Bug 2058879

Hello,

this is known limitation of discovery, discovery expects each subnet to be only in a single organization. You appear to have two:

2022-02-26T21:30:07 [I|app|0143a4af] Detected IPv4 subnet: satellite-subnet with taxonomy ["RedHat", "SCA"]/["GSS"]
2022-02-26T21:30:07 [I|app|0143a4af] Assigned location: GSS
2022-02-26T21:30:07 [I|app|0143a4af] Assigned organization: SCA

Then it's documented that Discovery will pick "the host uses the first organization and location associated with the subnet". Unfortunately, it is not able to specify which should be the one, discovery takes the "first" which is returned by the database. That is essentially random. We know it's limiting and terrible design, discovery would need substantial changes. There are currently no plans to fix this short-term.

---

7.3. Automatic Contexts for Discovered Hosts

Foreman server assigns an organization and location to discovered hosts according to the following sequence of rules:

If a discovered host uses a subnet defined in Foreman, the host uses the first organization and location associated with the subnet.

If the discovery_organization or discovery_location **fact** values are set, the discovered host uses these fact values as an organization and location. To set these fact values, navigate to Administer > Settings > Discovered, and add these facts to the Default organization and Default location fields. Ensure that the discovered host’s subnet also belongs to the organization and location set by the fact, otherwise Foreman refuses to set it for security reasons.

If none of the previous conditions exists, Foreman assigns the first Organization and Location ordered by name.

You can change the organization or location using the bulk actions menu of the Discovered hosts page. Select the discovered hosts to modify and select Assign Organization or Assign Location from the Select Action menu.

Note that the foreman_organization and foreman_location facts are no longer valid values for assigning context for discovered hosts. You still can use these facts to configure the host for Puppet runs. To do this, navigate to Administer > Settings > Puppet section and add the foreman_organization and foreman_location facts to the Default organization and Default location fields.

---

https://docs.theforeman.org/nightly/Provisioning_Guide/index-foreman-el.html#Automatic_Contexts_for_Discovered_Hosts_provisioning

The errors reported are not relevant to this problem, discovery just imports that other NICs had no subnets detected. The primary NIC was properly detected. To avoid these errors, create subnets for all your other networks.

Feel free to reopen if you think this is a different issue.

(In reply to Lukas Zapletal from comment #3)
> Hello,
> 
> this is known limitation of discovery, discovery expects each subnet to be
> only in a single organization. You appear to have two:
> 
> 2022-02-26T21:30:07 [I|app|0143a4af] Detected IPv4 subnet: satellite-subnet
> with taxonomy ["RedHat", "SCA"]/["GSS"]
> 2022-02-26T21:30:07 [I|app|0143a4af] Assigned location: GSS
> 2022-02-26T21:30:07 [I|app|0143a4af] Assigned organization: SCA
> 
> Then it's documented that Discovery will pick "the host uses the first
> organization and location associated with the subnet". Unfortunately, it is
> not able to specify which should be the one, discovery takes the "first"
> which is returned by the database. That is essentially random. We know it's
> limiting and terrible design, discovery would need substantial changes.
> There are currently no plans to fix this short-term.
> 
> ---
> 
> 7.3. Automatic Contexts for Discovered Hosts
> 
> Foreman server assigns an organization and location to discovered hosts
> according to the following sequence of rules:
> 
> If a discovered host uses a subnet defined in Foreman, the host uses the
> first organization and location associated with the subnet.
> 
> If the discovery_organization or discovery_location **fact** values are set,
> the discovered host uses these fact values as an organization and location.
> To set these fact values, navigate to Administer > Settings > Discovered,
> and add these facts to the Default organization and Default location fields.
> Ensure that the discovered host’s subnet also belongs to the organization
> and location set by the fact, otherwise Foreman refuses to set it for
> security reasons.
> 
> If none of the previous conditions exists, Foreman assigns the first
> Organization and Location ordered by name.
> 
> You can change the organization or location using the bulk actions menu of
> the Discovered hosts page. Select the discovered hosts to modify and select
> Assign Organization or Assign Location from the Select Action menu.
> 
> Note that the foreman_organization and foreman_location facts are no longer
> valid values for assigning context for discovered hosts. You still can use
> these facts to configure the host for Puppet runs. To do this, navigate to
> Administer > Settings > Puppet section and add the foreman_organization and
> foreman_location facts to the Default organization and Default location
> fields.
> 
> ---
> 
> https://docs.theforeman.org/nightly/Provisioning_Guide/index-foreman-el.
> html#Automatic_Contexts_for_Discovered_Hosts_provisioning
> 
> The errors reported are not relevant to this problem, discovery just imports
> that other NICs had no subnets detected. The primary NIC was properly
> detected. To avoid these errors, create subnets for all your other networks.
> 
> Feel free to reopen if you think this is a different issue.


Hello Lukas,

You are right about this part i.e. initially I had two Orgs associated with the same subnet.

2022-02-26T21:30:07 [I|app|0143a4af] Detected IPv4 subnet: satellite-subnet with taxonomy ["RedHat", "SCA"]/["GSS"]
2022-02-26T21:30:07 [I|app|0143a4af] Assigned location: GSS
2022-02-26T21:30:07 [I|app|0143a4af] Assigned organization: SCA


But I have mentioned also that, After I removed the SCA org from that Subnet , It still complains i.e.


2022-02-26T21:39:03 [I|app|75ff7319] Detected IPv4 subnet: satellite-subnet with taxonomy ["RedHat"]/["GSS"]
2022-02-26T21:39:03 [I|app|75ff7319] Assigned location: GSS
2022-02-26T21:39:03 [I|app|75ff7319] Assigned organization: SCA
2022-02-26T21:39:03 [W|app|75ff7319] Not queueing Nic::Managed: ["Subnet is not defined for host's organization"]
2022-02-26T21:39:03 [W|app|75ff7319] Not queueing Nic::Managed: ["Subnet is not defined for host's organization"]
2022-02-26T21:39:03 [W|app|75ff7319] Not queueing Nic::Managed: ["Subnet is not defined for host's organization"]
..
..
ActiveRecord::RecordInvalid): Validation failed: Interfaces.subnet is not defined for host's organization
 75ff7319 | /usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/validations.rb:80:in `raise_validation_error'
 75ff7319 | /usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/validations.rb:53:in `save!'



So at this point, This is valid but the Org is being assigned is the second one which is no longer part of subnet
~~
If the discovery_organization or discovery_location **fact** values are set, the discovered host uses these fact values as an organization and location. To set these fact values, navigate to Administer > Settings > Discovered, and add these facts to the Default organization and Default location fields. Ensure that the discovered host’s subnet also belongs to the organization and location set by the fact, otherwise Foreman refuses to set it for security reasons.
~~

So any thoughts here ?

Apart from subnet and the "Default Discovery Org\Loc" settings, what other facts could determine the selection of Discovery Org ? 


And about,
~~
You can change the organization or location using the bulk actions menu of the Discovered hosts page. Select the discovered hosts to modify and select Assign Organization or Assign Location from the Select Action menu.
~~

That's workable for manual deployments but when discovery rules are in place and Auto-Deployment is enabled, We will face problems here i.e. The Host will be created and Discovered on the Wrong Org.

This is the code:

        Organization.find_by_title(facts["discovery_organization"]) ||
          Organization.find_by_title(Setting[:discovery_organization]) ||
          host.subnet.try(:organizations).try(:first) ||
          host.subnet6.try(:organizations).try(:first) ||
          Organization.first

So it assigns:

- org from fact (not your case)
- org from settings (you have "RedHat" there I assume)
- org from subnet4 randomly (if setting is blank)
- org from subnet6 randomly (if setting is blank)
- randomly any org

Can you share exactly how your organization is entitled and named? What is exactly the title and what label?

So what you are seeing is misbehavior if you really have "RedHat" in the Discovery Organization. Could you please find and patch this file on your instance:

diff --git a/app/services/foreman_discovery/import_hooks/subnet_and_taxonomy.rb b/app/services/foreman_discovery/import_hooks/subnet_and_taxonomy.rb
index 8c6bb03..26ddd2b 100644
--- a/app/services/foreman_discovery/import_hooks/subnet_and_taxonomy.rb
+++ b/app/services/foreman_discovery/import_hooks/subnet_and_taxonomy.rb
@@ -46,6 +46,13 @@ module ForemanDiscovery

       def suggested_organization
         logger.warn("Do not use 'foreman_organization' fact for discovery and use 'discovery_organization' instead") if facts["foreman_organization"]
+
+        logger.warn("F " + Organization.find_by_title(facts["discovery_organization"]))
+        logger.warn("ST " + Setting[:discovery_organization])
+        logger.warn("SO " + Organization.find_by_title(Setting[:discovery_organization]))
+        logger.warn("S4 " + host.subnet.try(:organizations).try(:first))
+        logger.warn("S6 " + host.subnet6.try(:organizations).try(:first))
+        logger.warn("L " + Organization.first)
+
         Organization.find_by_title(facts["discovery_organization"]) ||
           Organization.find_by_title(Setting[:discovery_organization]) ||
           host.subnet.try(:organizations).try(:first) ||

Then restart "foreman" service and grab those logging lines, I wonder what you actually have in those settings.

Few initial checks:

[root@breakfix0-sat ~]# hammer settings info --name discovery_location
Id:            discovery_location
Name:          discovery_location
Description:   The default location to place discovered hosts in
Category:      Discovered
Settings type: string
Value:         GSS

[root@breakfix0-sat ~]# hammer settings info --name discovery_organization
Id:            discovery_organization
Name:          discovery_organization
Description:   The default organization to place discovered hosts in
Category:      Discovered
Settings type: string
Value:         RedHat



Info related to settings from Rake console:

irb(main):008:0> pp Setting.find_by_name('discovery_organization').default
D, [2022-03-05T01:25:12.458457 #415587] DEBUG -- :   Setting Load (0.7ms)  SELECT "settings".* FROM "settings" WHERE "settings"."name" = $1 ORDER BY "settings"."name" ASC LIMIT $2  [["name", "discovery_organization"], ["LIMIT", 1]]
""
=> ""

irb(main):009:0> pp Setting.find_by_name('discovery_organization').value
D, [2022-03-05T01:25:17.914090 #415587] DEBUG -- :   Setting Load (0.7ms)  SELECT "settings".* FROM "settings" WHERE "settings"."name" = $1 ORDER BY "settings"."name" ASC LIMIT $2  [["name", "discovery_organization"], ["LIMIT", 1]]
"RedHat"
=> "RedHat"

irb(main):010:0> pp Setting.find_by_name('discovery_location').default
D, [2022-03-05T01:25:26.751219 #415587] DEBUG -- :   Setting Load (1.0ms)  SELECT "settings".* FROM "settings" WHERE "settings"."name" = $1 ORDER BY "settings"."name" ASC LIMIT $2  [["name", "discovery_location"], ["LIMIT", 1]]
""
=> ""

irb(main):011:0> pp Setting.find_by_name('discovery_location').value
D, [2022-03-05T01:25:31.602879 #415587] DEBUG -- :   Setting Load (0.6ms)  SELECT "settings".* FROM "settings" WHERE "settings"."name" = $1 ORDER BY "settings"."name" ASC LIMIT $2  [["name", "discovery_location"], ["LIMIT", 1]]
"GSS"
=> "GSS"

irb(main):012:0> pp Setting["discovery_organization"]
"RedHat"
=> "RedHat"

irb(main):013:0> pp Setting["discovery_location"]
"GSS"
=> "GSS"

irb(main):003:0> pp Organization.first
D, [2022-03-05T01:40:47.533958 #415866] DEBUG -- :   Organization Load (0.8ms)  SELECT "taxonomies".* FROM "taxonomies" WHERE "taxonomies"."type" = $1 ORDER BY "taxonomies"."title" ASC LIMIT $2  [["type", "Organization"], ["LIMIT", 1]]
#<Organization:0x0000564310fc79f8
 id: 1,
 name: "RedHat",
 type: "Organization",
 created_at: Wed, 16 Feb 2022 11:51:21 UTC +00:00,
 updated_at: Sat, 19 Feb 2022 07:01:54 UTC +00:00,
 ignore_types: [],
 description: nil,
 label: "RedHat",
 ancestry: nil,
 title: "RedHat",
 manifest_refreshed_at: Sat, 19 Feb 2022 07:01:54 UTC +00:00,
 created_in_katello: true>
=> #<Organization id: 1, name: "RedHat", type: "Organization", created_at: "2022-02-16 11:51:21", updated_at: "2022-02-19 07:01:54", ignore_types: [], description: nil, label: "RedHat", ancestry: nil, title: "RedHat", manifest_refreshed_at: "2022-02-19 07:01:54", created_in_katello: true>



Now, without any changes and Subnet being in Org RedHat and Location GSS, 


2022-03-05T01:34:02 [I|aud|52d4ecc5] Host::Base (25) create event on otp 
2022-03-05T01:34:02 [I|aud|52d4ecc5] Host::Base (25) create event on realm_id 
2022-03-05T01:34:02 [I|aud|52d4ecc5] Host::Base (25) create event on compute_profile_id 
2022-03-05T01:34:02 [I|aud|52d4ecc5] Host::Base (25) create event on provision_method 
2022-03-05T01:34:02 [I|aud|52d4ecc5] Host::Base (25) create event on grub_pass 
2022-03-05T01:34:02 [I|aud|52d4ecc5] Host::Base (25) create event on discovery_rule_id 
2022-03-05T01:34:02 [I|aud|52d4ecc5] Host::Base (25) create event on global_status 0
2022-03-05T01:34:02 [I|aud|52d4ecc5] Host::Base (25) create event on lookup_value_matcher 
2022-03-05T01:34:02 [I|aud|52d4ecc5] Host::Base (25) create event on openscap_proxy_id 
2022-03-05T01:34:02 [I|aud|52d4ecc5] Host::Base (25) create event on pxe_loader 
2022-03-05T01:34:02 [I|aud|52d4ecc5] Host::Base (25) create event on initiated_at 
2022-03-05T01:34:02 [I|aud|52d4ecc5] Host::Base (25) create event on build_errors 
2022-03-05T01:34:03 [I|app|52d4ecc5] Import facts for 'mac005056b46ed6' completed. Added: 306, Updated: 0, Deleted 0 facts
2022-03-05T01:34:03 [I|aud|52d4ecc5] Nic::Managed (28) update event on mac , 00:50:56:b4:6e:d6
2022-03-05T01:34:03 [I|aud|52d4ecc5] Nic::Managed (28) update event on identifier , ens32
2022-03-05T01:34:03 [I|aud|52d4ecc5] Nic::Managed (28) update event on execution false, true
2022-03-05T01:34:03 [I|app|52d4ecc5] Detected IPv4 subnet: satellite-subnet with taxonomy ["RedHat"]/["GSS"]
2022-03-05T01:34:03 [I|app|52d4ecc5] Assigned location: GSS
2022-03-05T01:34:03 [I|app|52d4ecc5] Assigned organization: SCA
2022-03-05T01:34:03 [W|app|52d4ecc5] Not queueing Nic::Managed: ["Subnet is not defined for host's organization"]
2022-03-05T01:34:03 [W|app|52d4ecc5] Not queueing Nic::Managed: ["Subnet is not defined for host's organization"]
2022-03-05T01:34:03 [W|app|52d4ecc5] Not queueing Nic::Managed: ["Subnet is not defined for host's organization"]
2022-03-05T01:34:03 [W|app|52d4ecc5] Host discovery failed,



Now, After applying your patch and restarting foreman and httpd service:

** Deleted the existing discovered host

** Rebooted the Host into discovery and got a very weird traceback :


2022-03-05T01:45:31 [I|aud|4acdd64d] Host::Base (26) create event on initiated_at 
2022-03-05T01:45:31 [I|aud|4acdd64d] Host::Base (26) create event on build_errors 
2022-03-05T01:45:33 [I|app|4acdd64d] Import facts for 'mac005056b46ed6' completed. Added: 306, Updated: 0, Deleted 0 facts
2022-03-05T01:45:33 [I|aud|4acdd64d] Nic::Managed (29) update event on mac , 00:50:56:b4:6e:d6
2022-03-05T01:45:33 [I|aud|4acdd64d] Nic::Managed (29) update event on identifier , ens32
2022-03-05T01:45:33 [I|aud|4acdd64d] Nic::Managed (29) update event on execution false, true
2022-03-05T01:45:33 [I|app|4acdd64d] Detected IPv4 subnet: satellite-subnet with taxonomy ["RedHat"]/["GSS"]
2022-03-05T01:45:33 [I|app|4acdd64d] Assigned location: GSS
2022-03-05T01:45:33 [I|aud|4acdd64d] Host::Base (26) update event on model_id , 1
2022-03-05T01:45:33 [I|aud|4acdd64d] Host::Base (26) update event on owner_id , 4
2022-03-05T01:45:33 [I|aud|4acdd64d] Host::Base (26) update event on owner_type , User
2022-03-05T01:45:33 [I|aud|4acdd64d] Host::Base (26) update event on organization_id , 1
2022-03-05T01:45:33 [I|aud|4acdd64d] Host::Base (26) update event on location_id , 2
2022-03-05T01:45:33 [I|aud|4acdd64d] Nic::Managed (29) update event on subnet_id , 1
..
..
ype"=>:foreman_discovery}' error (TypeError): no implicit conversion of Organization into String
 4acdd64d | /usr/share/gems/gems/foreman_discovery-19.0.2/app/services/foreman_discovery/import_hooks/subnet_and_taxonomy.rb:50:in `+'
 4acdd64d | /usr/share/gems/gems/foreman_discovery-19.0.2/app/services/foreman_discovery/import_hooks/subnet_and_taxonomy.rb:50:in `suggested_organization'
 4acdd64d | /usr/share/gems/gems/foreman_discovery-19.0.2/app/services/foreman_discovery/import_hooks/subnet_and_taxonomy.rb:43:in `set_organization'
 4acdd64d | /usr/share/gems/gems/foreman_discovery-19.0.2/app/services/foreman_discovery/import_hooks/subnet_and_taxonomy.rb:18:in `after_populate'
 4acdd64d | /usr/share/gems/gems/foreman_discovery-19.0.2/app/services/foreman_discovery/import_hook_service.rb:20:in `block in after_populate'
 4acdd64d | /usr/share/gems/gems/foreman_discovery-19.0.2/app/services/foreman_discovery/import_hook_service.rb:18:in `each'
 4acdd64d | /usr/share/gems/gems/foreman_discovery-19.0.2/app/services/foreman_discovery/import_hook_service.rb:18:in `after_populate'
 4acdd64d | /usr/share/gems/gems/foreman_discovery-19.0.2/app/models/host/discovered.rb:132:in `populate_discovery_fields_from_facts'
 4acdd64d | /usr/share/gems/gems/foreman_discovery-19.0.2/app/models/host/discovered.rb:126:in `populate_fields_from_facts'
 4acdd64d | /usr/share/foreman/app/services/host_fact_importer.rb:50:in `block (2 levels) in parse_facts'
 4acdd64d | /usr/share/foreman/app/services/foreman/telemetry_helper.rb:28:in `telemetry_duration_histogram'
 4acdd64d | /usr/share/foreman/app/services/host_fact_importer.rb:49:in `block in parse_facts'
 4acdd64d | /usr/share/foreman/app/services/host_fact_importer.rb:93:in `skipping_orchestration'
 4acdd64d | /usr/share/foreman/app/services/host_fact_importer.rb:45:in `parse_facts'
 4acdd64d | /usr/share/foreman/app/services/host_fact_importer.rb:34:in `import_facts'
 4acdd64d | /usr/share/gems/gems/foreman_discovery-19.0.2/app/services/foreman_discovery/host_fact_importer.rb:8:in `import_facts'
 4acdd64d | /usr/share/gems/gems/foreman_discovery-19.0.2/app/models/host/discovered.rb:102:in `import_host'
 4acdd64d | /usr/share/gems/gems/foreman_discovery-19.0.2/app/controllers/api/v2/discovered_hosts_controller.rb:110:in `block in facts'
 4acdd64d | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'


If we see closely "Assigned organization" was never logged but when I went to GUI, I could see this entry in Discovered host despite the fact that discovery had failed:

Name	Model	IP Address	CPUs	Memory	Disk Count	Disks Size	Location	Organization	Subnet	Last Facts Upload	Actions
	 mac005056b46ed6	VMware Virtual Platform	192.168.230.102	1	3.86 GB	2	11 GB	GSS	RedHat	satellite-subnet (192.168.230.0/24)	18 minutes ago	



** Now, definitely the changes with logger you had mentioned was causing something odd so I used them it a bit different way:

diff --git a/app/services/foreman_discovery/import_hooks/subnet_and_taxonomy.rb b/app/services/foreman_discovery/import_hooks/subnet_and_taxonomy.rb
index 8c6bb03..4235e14 100644
--- a/app/services/foreman_discovery/import_hooks/subnet_and_taxonomy.rb
+++ b/app/services/foreman_discovery/import_hooks/subnet_and_taxonomy.rb
@@ -46,6 +46,14 @@ module ForemanDiscovery
 
       def suggested_organization
         logger.warn("Do not use 'foreman_organization' fact for discovery and use 'discovery_organization' instead") if facts["foreman_organization"]
+
+        logger.info "F : #{Organization.find_by_title(facts["discovery_organization"])}"
+        logger.info "ST : #{Setting[:discovery_organization]}"
+        logger.info "SO : #{Organization.find_by_title(Setting[:discovery_organization])}"
+        logger.info "S4 : #{host.subnet.try(:organizations).try(:first)}"
+        logger.info "S6 : #{host.subnet6.try(:organizations).try(:first)}"
+        logger.info "L : #{Organization.first}"
+
         Organization.find_by_title(facts["discovery_organization"]) ||
           Organization.find_by_title(Setting[:discovery_organization]) ||
           host.subnet.try(:organizations).try(:first) ||



And after restarting foreman and httpd service:

** Deleted the existing discovered host

** Rebooted the Host into discovery and got expected traceback :


2022-03-05T02:30:12 [I|aud|3e7c2508] Host::Base (28) create event on realm_id 
2022-03-05T02:30:12 [I|aud|3e7c2508] Host::Base (28) create event on compute_profile_id 
2022-03-05T02:30:12 [I|aud|3e7c2508] Host::Base (28) create event on provision_method 
2022-03-05T02:30:12 [I|aud|3e7c2508] Host::Base (28) create event on grub_pass 
2022-03-05T02:30:12 [I|aud|3e7c2508] Host::Base (28) create event on discovery_rule_id 
2022-03-05T02:30:12 [I|aud|3e7c2508] Host::Base (28) create event on global_status 0
2022-03-05T02:30:12 [I|aud|3e7c2508] Host::Base (28) create event on lookup_value_matcher 
2022-03-05T02:30:12 [I|aud|3e7c2508] Host::Base (28) create event on openscap_proxy_id 
2022-03-05T02:30:12 [I|aud|3e7c2508] Host::Base (28) create event on pxe_loader 
2022-03-05T02:30:12 [I|aud|3e7c2508] Host::Base (28) create event on initiated_at 
2022-03-05T02:30:12 [I|aud|3e7c2508] Host::Base (28) create event on build_errors 
2022-03-05T02:30:12 [W|app|] Creating scope :exportable. Overwriting existing method Katello::Repository.exportable.
2022-03-05T02:30:14 [I|app|3e7c2508] Import facts for 'mac005056b46ed6' completed. Added: 306, Updated: 0, Deleted 0 facts
2022-03-05T02:30:14 [I|aud|3e7c2508] Nic::Managed (31) update event on mac , 00:50:56:b4:6e:d6
2022-03-05T02:30:14 [I|aud|3e7c2508] Nic::Managed (31) update event on identifier , ens32
2022-03-05T02:30:14 [I|aud|3e7c2508] Nic::Managed (31) update event on execution false, true
2022-03-05T02:30:14 [I|app|3e7c2508] Detected IPv4 subnet: satellite-subnet with taxonomy ["RedHat"]/["GSS"]
2022-03-05T02:30:14 [I|app|3e7c2508] Assigned location: GSS
2022-03-05T02:30:14 [I|app|3e7c2508] F : SCA
2022-03-05T02:30:14 [I|app|3e7c2508] ST : RedHat
2022-03-05T02:30:14 [I|app|3e7c2508] SO : RedHat
2022-03-05T02:30:14 [I|app|3e7c2508] S4 : RedHat
2022-03-05T02:30:14 [I|app|3e7c2508] S6 : 
2022-03-05T02:30:14 [I|app|3e7c2508] L : RedHat
2022-03-05T02:30:14 [I|app|3e7c2508] Assigned organization: SCA
2022-03-05T02:30:14 [W|app|3e7c2508] Not queueing Nic::Managed: ["Subnet is not defined for host's organization"]
2022-03-05T02:30:14 [W|app|3e7c2508] Not queueing Nic::Managed: ["Subnet is not defined for host's organization"]
2022-03-05T02:30:14 [W|app|3e7c2508] Not queueing Nic::Managed: ["Subnet is not defined for host's organization"]
2022-03-05T02:30:14 [W|app|3e7c2508] Host discovery failed, facts: {"os"=>{"name"=>"RedHat", "family"=>"RedHat", "release"=>{"full"=>"7.9", "major"=>"7", "minor"=>"9"}, "hardware"=>"x86_64", 
..
..
..
pe"=>:foreman_discovery}' error (ActiveRecord::RecordInvalid): Validation failed: Interfaces.subnet is not defined for host's organization
..
 3e7c2508 | /usr/share/gems/gems/foreman_discovery-19.0.2/app/models/host/discovered.rb:134:in `populate_discovery_fields_from_facts'
 3e7c2508 | /usr/share/gems/gems/foreman_discovery-19.0.2/app/models/host/discovered.rb:126:in `populate_fields_from_facts'
 3e7c2508 | /usr/share/foreman/app/services/host_fact_importer.rb:50:in `block (2 levels) in parse_facts'
 3e7c2508 | /usr/share/foreman/app/services/foreman/telemetry_helper.rb:28:in `telemetry_duration_histogram'
 3e7c2508 | /usr/share/foreman/app/services/host_fact_importer.rb:49:in `block in parse_facts'
 3e7c2508 | /usr/share/foreman/app/services/host_fact_importer.rb:93:in `skipping_orchestration'
 3e7c2508 | /usr/share/foreman/app/services/host_fact_importer.rb:45:in `parse_facts'
 3e7c2508 | /usr/share/foreman/app/services/host_fact_importer.rb:34:in `import_facts'
 3e7c2508 | /usr/share/gems/gems/foreman_discovery-19.0.2/app/services/foreman_discovery/host_fact_importer.rb:8:in `import_facts'
 3e7c2508 | /usr/share/gems/gems/foreman_discovery-19.0.2/app/models/host/discovered.rb:102:in `import_host'
 3e7c2508 | /usr/share/gems/gems/foreman_discovery-19.0.2/app/controllers/api/v2/discovered_hosts_controller.rb:110:in `block in facts'
 3e7c2508 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
 3e7c2508 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
 3e7c2508 | /usr/share/gems/gems/foreman_discovery-19.0.2/app/controllers/api/v2/discovered_hosts_controller.rb:109:in `facts'


So it seems the first condition i.e. Organization.find_by_title(facts["discovery_organization"]) is giving SCA as the value of Org and hence that is being assigned to the system.


And as expected even if discovery shows failed, the host is partially discovered:

Name	Model	IP Address	CPUs	Memory	Disk Count	Disks Size	Location	Organization	Subnet	Last Facts Upload	Actions
	 mac005056b46ed6	N/A	192.168.230.102	N/A	0 Bytes	N/A	0 Bytes				11 minutes ago	



I hope this helps.

So the problem is that in your case, your organization SCA has no title set:

irb(main):004:0> Organization.unscoped.pluck(:id, :name, :label, :title)
=> [[1, "RedHat", "RedHat", "RedHat"], [3, "SCA", "SCA", nil]]

Therefore it can't be found and it falls back. Title is a field that is not editable by user, it should have been set automatically when creating the org.

It looks like Satellite 7.0 (or the build you are testing against) is missing this patch

https://bugzilla.redhat.com/show_bug.cgi?id=2045854

It was only fixed upstream in 3.1.2 (https://github.com/theforeman/foreman/pull/9107) while you have 3.1.1 and I don't see this to be backported:

[root@breakfix0-sat ~]# rpm -q foreman
foreman-3.1.1.6-1.el8sat.noarch

So, 

A) The older hammer output, shows the label available for the SCA Org.

# hammer organization list
---|--------|--------|-------------|-------
ID | TITLE  | NAME   | DESCRIPTION | LABEL 
---|--------|--------|-------------|-------
3  | SCA    | SCA    |             | SCA   
1  | RedHat | RedHat |             | RedHat
---|--------|--------|-------------|-------


But the DB info had reflected otherwise i.e. SCA org originally had no Title set.


B) You had confirmed that you could see the Title, after you simply edited the SCA org and saved it. 


C) I re-tested the discovery and now it works just fine i.e. the value of "Organization.find_by_title(facts["discovery_organization"])" comes as blank and the Host gets assigned with RedHat Org as expected.


So I guess, https://bugzilla.redhat.com/show_bug.cgi?id=2045854 was indeed causing the issue and that fix should fix this regression as well but still, one thing is confusing here.


As per the testing from comment 6, The failure happens as Organization.find_by_title(facts["discovery_organization"])  returns a value SCA but is the search was happening by the title and that Org never had any title stored in DB, Then I would expect the search result to be blank but how come it was returning SCA as the Org name there .

If I compare it with the current working scenario, I actually see this i.e.

2022-03-07T20:39:16 [I|app|0b7cab34] Detected IPv4 subnet: satellite-subnet with taxonomy ["RedHat"]/["GSS"]
2022-03-07T20:39:16 [I|app|0b7cab34] Assigned location: GSS
2022-03-07T20:39:16 [I|app|0b7cab34] F : 
2022-03-07T20:39:16 [I|app|0b7cab34] ST : RedHat
2022-03-07T20:39:16 [I|app|0b7cab34] SO : RedHat
2022-03-07T20:39:16 [I|app|0b7cab34] S4 : RedHat
2022-03-07T20:39:16 [I|app|0b7cab34] S6 : 
2022-03-07T20:39:16 [I|app|0b7cab34] L : RedHat
2022-03-07T20:39:16 [I|app|0b7cab34] Assigned organization: RedHat


Which means the value of "Organization.find_by_title(facts["discovery_organization"]) " is now blank but earlier i.e before fixing the title part it was SCA. How exactly the find_by_title method is searching for the Org here ?

If facts['discovery_organization'] == nil

Organization.find_by_title(facts["discovery_organization"]) will return the first record that has title nil in the database.

I belive easiest fix would be not to lookup the Loc/Org for blank facts, even better would be running a migration that will fill the missing titles. Which will still be blank in the DB even after https://bugzilla.redhat.com/show_bug.cgi?id=2045854 as it did not fill them up in the DB, only prevented new Org/Locs from being created with blanks.

(In reply to Ondřej Ezr from comment #10)
> If facts['discovery_organization'] == nil
> 
> Organization.find_by_title(facts["discovery_organization"]) will return the
> first record that has title nil in the database.
> 
> I belive easiest fix would be not to lookup the Loc/Org for blank facts,
> even better would be running a migration that will fill the missing titles.
> Which will still be blank in the DB even after
> https://bugzilla.redhat.com/show_bug.cgi?id=2045854 as it did not fill them
> up in the DB, only prevented new Org/Locs from being created with blanks.


About,
~~
If facts['discovery_organization'] == nil

Organization.find_by_title(facts["discovery_organization"]) will return the first record that has title nil in the database.
~~

True and it should have been always true but for some reason, before fixing the title issue, it was always reflecting SCA as it's value as you can see from comment 6 and I am just trying to find out why. 

It's not a blocker anymore i.e. after we have fixed the Title part but I am a bit curious as to how a nil value was interpreted as SCA org anyway..

When the first search returns nil then it goes further in the statement eventually matching something. This is a weak point of discovery but rather than fixing this, we would like to focus on rearchitecting this from scratch.

(In reply to Lukas Zapletal from comment #12)
> When the first search returns nil then it goes further in the statement
> eventually matching something. This is a weak point of discovery but rather
> than fixing this, we would like to focus on rearchitecting this from scratch.

Ack. I guess I can say, with the Org title fixed, the issue no longer exists or is reproducible. 

i.e. if a second Org was created on an installation before the fix from https://bugzilla.redhat.com/show_bug.cgi?id=2045854 landed, then only the issue will be reproducible, or else it should not.

Sayan, based on the comment 13, can this one be closed? The linked BZ is verified already in 6.11, so this particular BZ should not be observable anymore. Could you please confirm/close?

(In reply to Marek Hulan from comment #15)
> Sayan, based on the comment 13, can this one be closed? The linked BZ is
> verified already in 6.11, so this particular BZ should not be observable
> anymore. Could you please confirm/close?

Hello Marek,

Good Morning.

Yes. As long as https://bugzilla.redhat.com/show_bug.cgi?id=2045854 gets fixed in GA of 6.11 and CU upgrades the satellite after that, I don't think they will hit this issue again.

It was unfortunate that I had created my Org on the snap where the issue was still present and then the snap was upgraded to a working one, where I tested the Foreman discovery and ran into the issue. 


-- Sayan

Closing based upon comment 15 & 16.  The referenced bug 2045854 has been VERIFIED on 6.11.

If there are any concerns, please re-open with appropriate details.