Bug 2001552

This is probably caused by the fact, host is in "build mode" during the registration. The facts should get populated automatically in 4 hours. This would be a valid improvement to trigger the refresh right after the registration. User can also adjust the host init template to schedule the facts refresh using `at` command (e.g. a minute after the registration)

I recently upgraded to Satellite 6.10 and thus started using the global registration form to register my pre-existing (i.e. not built by Satellite) machines.  I noticed hosts registered to the satellite were missing their facts while others registered manually were not.

The global registration template check for the existence of /etc/redhat-release.  If it finds it, then eventually these 2 commands are executed:

    subscription-manager register <%= '--force' if @force %> \
        --org='<%= @organization.label %>' \
        --activationkey=<%= shell_escape(activation_keys) %> || <%= @ignore_subman_errors ? 'true' : 'exit 1' %>

    register_katello_host | bash

In troubleshooting this, I discovered that the host gets registered just fine -- with facts -- after the execution of the 1st, "subscription-manager register" command.  But then the 2nd command -- the register_katello_host function -- removes the facts from the profile in the satellite.

@mhulan : you wrote above that that this is caused by the host being in "build mode" during registration and that facts should get populated in ~4 hours.  I do not think my hosts were in "build mode"; they were built via the RHEL DVD installer.  Also, no amount of waiting has caused the facts to be restored.

I should note that executing 'subscription-manager facts --update' on the host has no effect, despite the log files showing "success".  Ditto for 'subscription-manager refresh'.


FROM HOST
==> /var/log/rhsm/rhsm.log <==
2021-12-28 14:16:12,188 [DEBUG] subscription-manager:1618:MainThread @https.py:56 - Using standard libs to provide httplib and ssl
2021-12-28 14:16:12,410 [DEBUG] subscription-manager:1618:MainThread @dbus_interface.py:37 - self.has_main_loop=False
2021-12-28 14:16:12,433 [DEBUG] subscription-manager:1618:MainThread @ga_loader.py:91 - ga_loader GaImporterGtk3
2021-12-28 14:16:12,434 [DEBUG] subscription-manager:1618:MainThread @plugins.py:571 - loaded plugin modules: []
2021-12-28 14:16:12,435 [DEBUG] subscription-manager:1618:MainThread @plugins.py:572 - loaded plugins: {}
2021-12-28 14:16:12,435 [DEBUG] subscription-manager:1618:MainThread @identity.py:139 - Loading consumer info from identity certificates.
2021-12-28 14:16:12,439 [DEBUG] subscription-manager:1618:MainThread @connection.py:808 - Environment variable NO_PROXY= will be used
2021-12-28 14:16:12,439 [INFO] subscription-manager:1618:MainThread @connection.py:905 - Connection built: host=satellite.localdomain port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2021-12-28 14:16:12,440 [DEBUG] subscription-manager:1618:MainThread @files.py:322 - Successfully read local syspurpose contents.
2021-12-28 14:16:12,440 [DEBUG] subscription-manager:1618:MainThread @files.py:359 - Successfully read cached syspurpose contents.
2021-12-28 14:16:12,451 [DEBUG] subscription-manager:1618:MainThread @files.py:322 - Successfully read local syspurpose contents.
2021-12-28 14:16:12,452 [DEBUG] subscription-manager:1618:MainThread @files.py:359 - Successfully read cached syspurpose contents.
2021-12-28 14:16:12,456 [DEBUG] subscription-manager:1618:MainThread @files.py:322 - Successfully read local syspurpose contents.
2021-12-28 14:16:12,456 [DEBUG] subscription-manager:1618:MainThread @files.py:359 - Successfully read cached syspurpose contents.
2021-12-28 14:16:12,457 [DEBUG] subscription-manager:1618:MainThread @files.py:322 - Successfully read local syspurpose contents.
2021-12-28 14:16:12,457 [DEBUG] subscription-manager:1618:MainThread @files.py:359 - Successfully read cached syspurpose contents.
2021-12-28 14:16:12,458 [DEBUG] subscription-manager:1618:MainThread @managercli.py:501 - X-Correlation-ID: ad4c3fbd717f4ad89461741c81c14d9c
2021-12-28 14:16:12,458 [DEBUG] subscription-manager:1618:MainThread @managercli.py:390 - Client Versions: {'subscription-manager': '1.24.42-1.el7'}
2021-12-28 14:16:12,458 [DEBUG] subscription-manager:1618:MainThread @connection.py:808 - Environment variable NO_PROXY= will be used
2021-12-28 14:16:12,459 [INFO] subscription-manager:1618:MainThread @connection.py:905 - Connection built: host=satellite.localdomain port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2021-12-28 14:16:12,459 [DEBUG] subscription-manager:1618:MainThread @connection.py:808 - Environment variable NO_PROXY= will be used
2021-12-28 14:16:12,459 [INFO] subscription-manager:1618:MainThread @connection.py:905 - Connection built: host=satellite.localdomain port=443 handler=/rhsm auth=none
2021-12-28 14:16:12,460 [DEBUG] subscription-manager:1618:MainThread @managercli.py:366 - Consumer Identity name=rhel79-010.localdomain uuid=d94da141-950f-418d-b967-0977eef95acb
2021-12-28 14:16:12,460 [DEBUG] subscription-manager:1618:MainThread @cache.py:169 - Checking current system info against cache: /var/lib/rhsm/facts/facts.json
2021-12-28 14:16:12,484 [DEBUG] subscription-manager:1618:MainThread @dmiinfo.py:76 - Using dmidecode dump file: /dev/mem
2021-12-28 14:16:12,576 [DEBUG] subscription-manager:1618:MainThread @insights.py:68 - Unable to read insights machine_id file: /etc/insights-client/machine-id, error: [Errno 2] No such file or directory: '/etc/insights-client/machine-id'
2021-12-28 14:16:12,577 [DEBUG] subscription-manager:1618:MainThread @insights.py:68 - Unable to read insights machine_id file: /etc/redhat-access-insights/machine-id, error: [Errno 2] No such file or directory: '/etc/redhat-access-insights/machine-id'
2021-12-28 14:16:12,577 [DEBUG] subscription-manager:1618:MainThread @cache.py:171 - System data has changed, updating server.
2021-12-28 14:16:12,577 [DEBUG] subscription-manager:1618:MainThread @facts.py:87 - Updating facts on server
2021-12-28 14:16:12,579 [DEBUG] subscription-manager:1618:MainThread @connection.py:523 - Loaded CA certificates from /etc/rhsm/ca/: redhat-uep.pem, katello-server-ca.pem, katello-default-ca.pem
2021-12-28 14:16:12,580 [DEBUG] subscription-manager:1618:MainThread @connection.py:571 - Making request: PUT /rhsm/consumers/d94da141-950f-418d-b967-0977eef95acb
2021-12-28 14:16:12,919 [DEBUG] subscription-manager:1618:MainThread @connection.py:618 - Response: status=200, request="PUT /rhsm/consumers/d94da141-950f-418d-b967-0977eef95acb"
2021-12-28 14:16:12,921 [DEBUG] subscription-manager:1618:MainThread @cache.py:114 - Wrote cache: /var/lib/rhsm/facts/facts.json
2021-12-28 14:16:12,921 [DEBUG] subscription-manager:1618:MainThread @managercli.py:2137 - Succesfully updated the system facts.
2021-12-28 14:16:12,922 [DEBUG] subscription-manager:1618:MainThread @repolib.py:175 - The rhsm.auto_enable_yum_plugins is enabled
2021-12-28 14:16:12,922 [DEBUG] subscription-manager:1618:MainThread @repolib.py:151 - dnf plugin: "/etc/dnf/plugins/subscription-manager.conf" already enabled. Nothing to do.
2021-12-28 14:16:12,923 [DEBUG] subscription-manager:1618:MainThread @repolib.py:151 - dnf plugin: "/etc/dnf/plugins/product-id.conf" already enabled. Nothing to do.
2021-12-28 14:16:12,923 [DEBUG] subscription-manager:1618:MainThread @repolib.py:151 - yum plugin: "/etc/yum/pluginconf.d/subscription-manager.conf" already enabled. Nothing to do.
2021-12-28 14:16:12,923 [DEBUG] subscription-manager:1618:MainThread @repolib.py:151 - yum plugin: "/etc/yum/pluginconf.d/product-id.conf" already enabled. Nothing to do.




FROM SATELLITE
==> /var/log/foreman/production.log <==
2021-12-28T14:16:12 [I|app|5297d836] Started PUT "/rhsm/consumers/d94da141-950f-418d-b967-0977eef95acb" for 192.168.5.130 at 2021-12-28 14:16:12 -0500
2021-12-28T14:16:12 [I|app|5297d836] Processing by Katello::Api::Rhsm::CandlepinProxiesController#facts as JSON
2021-12-28T14:16:12 [I|app|5297d836]   Parameters: {"facts"=>"[FILTERED]", "id"=>"d94da141-950f-418d-b967-0977eef95acb"}
2021-12-28T14:16:12 [I|app|5297d836] Completed 200 OK in 248ms (Views: 0.4ms | ActiveRecord: 18.2ms | Allocations: 15857)

==> /var/log/httpd/foreman-ssl_access_ssl.log <==
192.168.5.130 - - [28/Dec/2021:14:16:12 -0500] "PUT /rhsm/consumers/d94da141-950f-418d-b967-0977eef95acb HTTP/1.1" 200 41 "-" "RHSM/1.0 (cmd=subscription-manager)"

My satellite has these Administer ---> Settings ---> Puppet tab settings:

  Create new host when facts are uploaded	= Yes

  Update environment from facts = No

  Update hostgroup from facts = Yes

  Update subnets from facts = None


Changing these:

  Update environment from facts = Yes

  Update subnets from facts = All

...per:

  [Satellite 6] Content Host facts not updating
  correctly in Satellite
  https://access.redhat.com/solutions/4017611

...had no effect after again running 'subscription-manager facts --update' and 'subscription-manager refresh'.

(In reply to Bernie Hoefer from comment #2)
===
> you wrote above that that this is caused by the host
> being in "build mode" during registration and that facts should get
> populated in ~4 hours.
===

I think I understand, now, what you were saying, Marek.  If I run the "hammer host info" command, my host's status is:

  Global Status: Warning
  Build Status:  Pending installation

I can clear that by 1 of 2 ways:

1. browsing to my satellite's Hosts ---> All Hosts ---> {click on host
   name} ---> {click "clear" on the Build row}.

2. running this curl command from the host:

   curl --silent --show-error \
   "https://satellite.localdomain/unattended/built" \
   -H 'Authorization: Bearer {token from original curl command}

Either way, that changes the output of my "hammer host info" command to show:

  Global Status: Error
  Build Status:  Installed

...and running "subscription-manager facts --update" after that causes the facts to now be displayed in the satellite.


Still, it seems to me that something is wrong with the global registration template if it leaves hosts in a pending installation state.

Yeah, I agree this feels broken. Bernie, in your case, it seems that the build status got stuck in pending installation for some reason though. If the registration finishes successfully, it should switch the host to Installed automatically as the very last step. You can find that curl in the "Linux host_init_config default" init template shipped with Satellite 6.10. It should basically do something like this


> # Call home to exit build mode
> 
> trap - ERR
> foreman_curl 'http://sat6-10.example.com/unattended/built'
> 
> if [[ $? == 0 ]] ; then
>   echo "Host [host.example.com] successfully configured."
> else
>   echo "Host [host.example.com] successfully configured, but failed to set built status."
> fi
> 
> exit 0

I'd start checking what you get from the initial curl command without pipeing it to the bash and check this part is there. If it is there but it didn't get executed, some step before it may have failed and that's why your host remains in Pending installation.

(In reply to Marek Hulan from comment #6)
===
> It should basically do something like this
> 
> 
> > # Call home to exit build mode
> > 
> > trap - ERR
> > foreman_curl 'http://sat6-10.example.com/unattended/built'
===

It looks like there is a problem with the foreman_url('built') function in Satellite's "Linux host_init_config default" template.  I state this because when it goes to end the build, I get this curl command:

  curl \
  --silent \
  --show-error \
  --cacert $SSL_CA_CERT \
  -o /dev/null \
  --noproxy \* \
  'https://satellite.localdomain:9090/unattended/built?token=873e260b-0ae4-408b-9e73-8374f603fcd2'

...which returns a "Requested url was not found" error.  If I change that curl command to remove the TCP port 9090 specification, then it works fine and my host's build status changes to "Installed".

Alright, that explains why the registration does not succeed. The foreman_url('built') is based on the selected Capsule or Unattended URL setting if no Capsule is selected.

Can your host talk directly to Satellite or dies it have to go through a Capsule? If it can, have you selected any Capsule in the registration form? If so, try again without selecting any.

If the traffic must be proxied through the Capsule, make sure it has both Registration and Templates feature enabled. For that see Infrastrcuture -> Capsules, refresh it's features. If you see both of them, make sure the foreman_url is correctly set on that Capsule by running

> grep foreman_url /etc/foreman-proxy/settings.yml

Hi

Being able to select Satellite URL as internal Capsule in the web UI and therefore getting port 9090 added has been addressed for 7.0 in this bug:

Bug 2014251 - Global Registration: Selecting Satellite URL as the proxy fails to register hosts with default config

(In reply to Marek Hulan from comment #8)
===
> Can your host talk directly to Satellite or dies it have to go through a Capsule?
===

My host talks directly to the satellite.


===
> If it can, have you selected any Capsule in the registration form?
===

Yes, on the "Register Host" web page, I choose my satellite from the Capsule drop-down.


===
> If so, try again without selecting any.
===

That allowed my host to be registered with its build status as "Installed".

*However*, the host still showed up with no facts.  Running "subscription-manager facts --update" fixed that.  (Or waiting ~4 hours, as mentioned in comment #1.)  I wonder, if in this process:

1. The "/register" curl command runs "subscription-manager"; host gets
   registered with facts.

2. A 2nd "/register" curl command gets run, causing the build status
   to go to "Pending Installation" and the facts to be erased.

3. An "/unattended/built" curl command gets run, causing the the build
   status to go back to "Installed" but leaving the facts blank.

...the "Linux host_init_config default" template shouldn't just include a "subscription-manager facts --update" command so customers aren't left wondering where the facts are.


===
> If the traffic must be proxied through the Capsule, make sure it has both Registration and Templates feature enabled.
===

You are right; my satellite's built-in capsule only has the registration feature.  It lacks the template feature.  Thank you, Marek!


(In reply to Stephen Wadeley from comment #9)
===
> Being able to select Satellite URL as internal Capsule in the web UI and therefore getting port 9090 added has been addressed for 7.0 in this bug:
> 
> Bug 2014251 - Global Registration: Selecting Satellite URL as the proxy fails to register hosts with default config
===

Thanks for the clue, Stephen.

> Yes, on the "Register Host" web page, I choose my satellite from the Capsule drop-down.

That actually means to use the internal Capsule, which is not necessary at all. It's still a possibility, but then the internal Capsule needs to be configured correctly for proxying the registration requests.

> ...the "Linux host_init_config default" template shouldn't just include a "subscription-manager facts --update" command so customers aren't left wondering where the facts are.

I agree. Or we should trigger facts scan right after the /unattended/built curl

Created redmine issue https://projects.theforeman.org/issues/34249 from this bug

Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/34249 has been resolved.

Version-Release number of selected component (if applicable): - Sat 6.12.0 snap 7

How reproducible:
- Always

Steps to Reproduce:
1. Import Manifest
2. Enable and sync a repository(Ansible Engine 2.9 for RHEL 7)
3. Create Content view, add the synced repo to CV and publish it to Library environment.
4. Create activation key, and enable said repo.
5. Go to Hosts > All Host > Click on Register Host button.
6. Generate command to register content host.
7. Register a content host.
8. Go to Hosts > All Hosts > Click on host name

Actual results:
- Facts are present for the host.

Expected results:
- "Facts" section is present for the registered host.

Additional Info: "Successfully updated the system facts." is in the log for registration on the host.
"

In case 03352365 affected by this issue, we were able to work around it (on a host-by-host basis) using two commands.  First we ran this command on the Satellite server:

# hammer host update --id $i --build false

Then we ran this command on the target host:

# subscription-manager facts --update

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.12 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8506