Bug 1739367

The interesting part is:
[RUNNING], [2019-08-09 02:57:04], [/usr/sbin/subscription-manager register --org 'Default_Organization' --name 'cloud-qe-16-vm-08.idmqe.lab.eng.bos.redhat.com' --activationkey 'RH00076_key'  --serverurl=https://hpe-dl580g7-02.hpe2.lab.eng.bos.redhat.com:443/rhsm --baseurl=https://hpe-dl580g7-02.hpe2.lab.eng.bos.redhat.com/pulp/repos --force]
HTTP error (422 - Unprocessable Entity): Please unregister or remove hosts which match this host before registering: cloud-qe-16-vm-08.idmqe.lab.eng.bos.redhat.com
[ERROR], [2019-08-09 02:57:10], EXITING: [/usr/sbin/subscription-manager register --org 'Default_Organization' --name 'cloud-qe-16-vm-08.idmqe.lab.eng.bos.redhat.com' --activationkey 'RH00076_key'  --serverurl=https://hpe-dl580g7-02.hpe2.lab.eng.bos.redhat.com:443/rhsm --baseurl=https://hpe-dl580g7-02.hpe2.lab.eng.bos.redhat.com/pulp/repos --force] failed to execute properly.

This is https://bugzilla.redhat.com/show_bug.cgi?id=1730146 (6.6) or https://bugzilla.redhat.com/show_bug.cgi?id=1728291 (6.5) and as such I don't think this is a bug in katello-client-bootstrap (but I might be biased).
Given that https://bugzilla.redhat.com/show_bug.cgi?id=1728291 is actually fixed in 6.5.2, this makes me wonder if the 6.5.2 fix is incomplete?

NEEDINFO on the fix author in 6.5.2 to verify.

Okay, so after looking a bit closer into this, I think this is a regression in Katello as shipped in Satellite 6.5.2.

The code in the Subscription Facet that is trying to find an existing host looks like this:
        def self.find_host(facts, organization)
          host_name = propose_existing_hostname(facts)
          host_uuid = facts['dmi.system.uuid']
          uuid_fact_id = RhsmFactName.find_by(name: 'dmi::system::uuid')&.id || -1
  
          hosts = ::Host.unscoped.distinct.left_outer_joins(:fact_values)
                  .where("#{::Host.table_name}.name = ? OR (#{FactValue.table_name}.fact_name_id = ?
                 AND #{FactValue.table_name}.value = ?)", host_name, uuid_fact_id, host_uuid)
  
          return if hosts.empty?
  
          hosts = hosts.where(organization_id: [organization.id, nil])
          hosts_size = hosts.size
  
          if hosts_size == 0 # not in the correct org
            #TODO: http://projects.theforeman.org/issues/11532
            fail Katello::Errors::RegistrationError, _("Host with name %{host_name} is currently registered to a different org, please migrate host to %{org_name}.") %
                     {:org_name => organization.name, :host_name => host_name }
          end
  
          if hosts_size == 1
            host = hosts.first
            found_uuid = host.fact_values.find { |fv| fv.fact_name_id == uuid_fact_id }
  
            return host if host.name == host_name && (host.build || found_uuid&.value == host_uuid)
          end
  
          hostnames = hosts.pluck(:name).sort.join(', ')
          fail Katello::Errors::RegistrationError, _("Please unregister or remove hosts which match this host before registering: %{existing}") % {existing: hostnames}
        end

Especially this is interesting:
            return host if host.name == host_name && (host.build || found_uuid&.value == host_uuid)

This means that it will only find the correct host if the name matches and either the host is in build mode or the host record has the same UUID as the registering host.

This sadly completely breaks the katello-client-bootstrap workflow which does create a host record via the API (so that the host lands in the right hostgroup etc), but neither sets build mode (because it should not build the host, the host already exists!) nor submits any facts.

Making bootstrap set build mode is impossible: it sometimes does not have enough data collected to convince Foreman that build is even possible, and even if it would collect enough data, the host already exists and depending on the compute resource of the hostgroup you can get funny results if you put the host into build mode.

Making bootstrap find and submit the UUID when creating the host *might* be an option, but would require teaching bootstrap how to fake sub-man facts directly (instead of relying on actual sub-man to submit them for us because that is too late).

Created redmine issue https://projects.theforeman.org/issues/27580 from this bug

Upstream PR opened. Thanks for the report.

Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/27580 has been resolved.

*** Satellite 6.5.2 Hotfix Available ***

1) Download tfm-rubygem-katello-3.10.0.54-3.HOTFIXBZ1739367.el7sat.noarch.rpm from this bugzilla to your Satellite

2) Install:

rpm -Uvh tfm-rubygem-katello-3.10.0.54-3.HOTFIXBZ1739367.el7sat.noarch.rpm

3) restart:

satellite-maintain service restart

4) resume operations

Created attachment 1603497 [details]
tfm-rubygem-katello-3.10.0.54-3.HOTFIXBZ1739367.el7sat.noarch.rpm

I am seeing the same issue without pre-registering.

host build=yes
subscription-manager register --org=X --activationkey=AK1 --force   -> OK

host build=no
subscription-manager register --org=X --activationkey=AK1 --force   -> FAILURE

No deletion of host record done, nor an UUID change on the host itself

At least the fix of https://projects.theforeman.org/issues/27580 fixes it behavoir also. If this fix fixed also the issue with the host not changing the uuid and the host was already preregistered in sat6 the it looks also to be an issue that the host record is not storing the uuid correctly or the previous check was not loading the correct fields.

I troubleshooted the issue:

host build=yes
subscription-manager register --org=X --activationkey=AK1 --force   -> OK
fact system::dmi::uuid updated    -> FAIL, the value is not updated in the host


This make the behaviour the same as the API.

That the uuid (that has become an important field) is not saved during build=yes is in my opinion also a bug and might need to be tracked separate.

Hi Peter,

How did you go about ensuring that the dmi.system.uuid fact would be changed with doing subscription-manager register? I want to make sure I am following before I look into this possible bug. Any successful registration will trigger an update of the facts so I'm quite curious about your report.

Setting build=yes on the host will delete all facts including the uuid.

----
[crash/LI] root@li-lc-2222:~# hammer -c /opt/hoici/etc/sat6/hammer-hoici.yaml host facts --name li-lc-2231.hag.hilti.com | grep 'dmi::system::uuid'
dmi::system::uuid                                                 | 4217fee5-2f68-c100-1972-721cdf2fa3e1                                      
[crash/LI] root@li-lc-2222:~# hammer -c /opt/hoici/etc/sat6/hammer-hoici.yaml host update --name li-lc-2231.hag.hilti.com --managed yes --build yes
Host updated.
[crash/LI] root@li-lc-2222:~# hammer -c /opt/hoici/etc/sat6/hammer-hoici.yaml host facts --name li-lc-2231.hag.hilti.com | grep 'dmi::system::uuid'
#
# Now on client li-lc-2231
# 
[crash/LI] root@li-lc-2231:~# subscription-manager register --org=Hilti --activationkey=hg-crash::LI::CITest::CITest-6.10 --force
Unregistering from: fibssat6sb.hag.hilti.com:443/rhsm
The system with UUID c1c3a4eb-e17a-4d23-8ef2-6364d68554c2 has been unregistered
All local data removed
The system has been registered with ID: f6b0affe-66bf-4503-bfee-2f52d5c4f14e
The registered system name is: li-lc-2231
Installed Product Current Status:
...
#
# Back to Sat6
#
[crash/LI] root@li-lc-2222:~# hammer -c /opt/hoici/etc/sat6/hammer-hoici.yaml host facts --name li-lc-2231.hag.hilti.com | grep 'dmi::system::uuid'
[crash/LI] root@li-lc-2222:~#
-----

If the uuid is that important i think it should be part of the host record directly and not be part of the dynamic facts. if uuid is part of host record the delete of facts with build=yes would also not be an issue

(In reply to Peter Vreman from comment #29)
> Setting build=yes on the host will delete all facts including the uuid.

Correct
> 
> ----
> [crash/LI] root@li-lc-2222:~# hammer -c
> /opt/hoici/etc/sat6/hammer-hoici.yaml host facts --name
> li-lc-2231.hag.hilti.com | grep 'dmi::system::uuid'
> dmi::system::uuid                                                 |
> 4217fee5-2f68-c100-1972-721cdf2fa3e1                                      
> [crash/LI] root@li-lc-2222:~# hammer -c
> /opt/hoici/etc/sat6/hammer-hoici.yaml host update --name
> li-lc-2231.hag.hilti.com --managed yes --build yes
> Host updated.
> [crash/LI] root@li-lc-2222:~# hammer -c
> /opt/hoici/etc/sat6/hammer-hoici.yaml host facts --name
> li-lc-2231.hag.hilti.com | grep 'dmi::system::uuid'
> #
> # Now on client li-lc-2231
> # 
> [crash/LI] root@li-lc-2231:~# subscription-manager register --org=Hilti
> --activationkey=hg-crash::LI::CITest::CITest-6.10 --force
> Unregistering from: fibssat6sb.hag.hilti.com:443/rhsm
> The system with UUID c1c3a4eb-e17a-4d23-8ef2-6364d68554c2 has been
> unregistered
> All local data removed
> The system has been registered with ID: f6b0affe-66bf-4503-bfee-2f52d5c4f14e
> The registered system name is: li-lc-2231
> Installed Product Current Status:
> ...
> #
> # Back to Sat6
> #
> [crash/LI] root@li-lc-2222:~# hammer -c
> /opt/hoici/etc/sat6/hammer-hoici.yaml host facts --name
> li-lc-2231.hag.hilti.com | grep 'dmi::system::uuid'
> [crash/LI] root@li-lc-2222:~#
> -----
> 
> If the uuid is that important i think it should be part of the host record
> directly and not be part of the dynamic facts. if uuid is part of host
> record the delete of facts with build=yes would also not be an issue

I see the behavior you're illustrating but I'm not sure there's a problem. If one sets a host to build=true it's to indicate that the host will be rebuilt with a fresh OS at which time the system will be registered by the kickstart process or later on by hand. Registering the host in this window of time is unusual in my opinion. By the time the build process completes the facts will be present and what is important is that the facts are received by the registration process which they are. In other words: manually setting the host into build mode to work around registration problems is not recommended as that's not the intention of the build flag.

Also, I think it only makes sense to have the DMI UUID added as a field on the host record itself if that field is also unique. That would have broken environments which had hosts reporting duplicate DMI UUIDs prior to us making its uniqueness enforced in 6.5. We aren't at that point yet but perhaps in the future.

The manual setting of build=yes in my example was just the simple reproducer.

The statement 'By the time the build process completes the facts will be present' is sunny-day scenario only. In real-world, especially during engineering and testing, the kickstart sometimes fails and then facts (dmi-uuid) are not available on the host record.
Also I do not see the why a hosts.dmi-uuid field must be unique. Currently there is also no unique in the facts. And in the current checks only the combination hostname+dmi-uuid is verified to be unique.

I'd like to mention that when I installed a new RHEL 7.7 box in my test bed yesterday, it was also not subscribed correctly to RHSAT for - yet - unknown reasons, and therefore I ran into the very same issue, that I wanted to register the host, but sub-man didn't let me do that; Updating the host on RHSAT with build=yes was a work-around to get it registered.

This is the tracker for Satellite 6.6 and this issue has been fixed in Satellite 6.5.2.1 -> https://bugzilla.redhat.com/show_bug.cgi?id=1743351 Please do upgrade as soon as possible.

Verified on RHEL-8.1.0-20190821.0 register against Satellite 6.5.2.1.
Failed for 
[NOTIFICATION], [2019-08-23 02:37:57], [Writing FQDN katello-fact]
[ERROR], [2019-08-23 02:37:57], EXITING: [0 element in array for search key 'title="hg"' in API '/api/v2/hostgroups'. Please note that all searches are case-sensitive.] failed to execute properly.
[ERROR], [2019-08-23 02:37:57], EXITING: [Please also ensure that the user has permissions to view the searched objects. Fatal error.] failed to execute properly.


subscription-manager-1.25.14-1.el8.x86_64
dnf-plugin-subscription-manager-1.25.14-1.el8.x86_64
subscription-manager-cockpit-1.25.14-1.el8.noarch
python3-subscription-manager-rhsm-1.25.14-1.el8.x86_64
subscription-manager-rhsm-certificates-1.25.14-1.el8.x86_64
tfm-rubygem-katello-3.10.0.55-1.el7sat.noarch
katello-3.10.0-0.6.rc1.el7sat.noarch
candlepin-2.5.17-1.el7sat.noarch


[root@cloud-qe-16-vm-07 ~]# python3 /root/bootstrap.py --login=admin --password=admin --server=hpe-dl580g7-02.hpe2.lab.eng.bos.redhat.com --location='Default Location' --organization='Default Organization' --hostgroup=hg --activationkey=RH00076_key --add-domain --force
Foreman Bootstrap Script
This script is designed to register new systems or to migrate an existing system to a Foreman server with Katello
[NOTIFICATION], [2019-08-23 02:37:45], [This system is not registered to RHN. Attempting to register via subscription-manager]
[NOTIFICATION], [2019-08-23 02:37:45], [Checking subscription manager prerequisites]
[RUNNING], [2019-08-23 02:37:45], [/usr/bin/yum -y remove subscription-manager-gnome]
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
No match for argument: subscription-manager-gnome
No packages marked for removal.
Dependencies resolved.
Nothing to do.
Complete!
[SUCCESS], [2019-08-23 02:37:46], [/usr/bin/yum -y remove subscription-manager-gnome], completed successfully.
[NOTIFICATION], [2019-08-23 02:37:46], [subscription-manager is installed already. Attempting update]
[RUNNING], [2019-08-23 02:37:46], [/usr/bin/yum -y update subscription-manager 'subscription-manager-migration-*']
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Last metadata expiration check: 0:01:36 ago on Fri 23 Aug 2019 02:36:12 AM EDT.
Dependencies resolved.
Nothing to do.
Complete!
[SUCCESS], [2019-08-23 02:37:48], [/usr/bin/yum -y update subscription-manager 'subscription-manager-migration-*'], completed successfully.
[RUNNING], [2019-08-23 02:37:48], [/usr/bin/yum -y update yum openssl python]
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Last metadata expiration check: 0:01:38 ago on Fri 23 Aug 2019 02:36:12 AM EDT.
No match for argument: python
Dependencies resolved.
Nothing to do.
Complete!
[SUCCESS], [2019-08-23 02:37:51], [/usr/bin/yum -y update yum openssl python], completed successfully.
[NOTIFICATION], [2019-08-23 02:37:51], [Removing old Katello agent and certs]
[RUNNING], [2019-08-23 02:37:51], [/usr/bin/yum -y remove 'katello-ca-consumer-*' katello-agent gofer katello-host-tools katello-host-tools-fact-plugin]
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
No match for argument: katello-agent
No match for argument: gofer
No match for argument: katello-host-tools
No match for argument: katello-host-tools-fact-plugin
Dependencies resolved.
================================================================================
 Package                                             Arch   Version
                                                                  Repo     Size
================================================================================
Removing:
 katello-ca-consumer-hpe-dl580g7-02.hpe2.lab.eng.bos.redhat.com
                                                     noarch 1.0-1 @System  16 k

Transaction Summary
================================================================================
Remove  1 Package

Freed space: 16 k
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Erasing          : katello-ca-consumer-hpe-dl580g7-02.hpe2.lab.eng.bos.   1/1 
  Running scriptlet: katello-ca-consumer-hpe-dl580g7-02.hpe2.lab.eng.bos.   1/1 
  Verifying        : katello-ca-consumer-hpe-dl580g7-02.hpe2.lab.eng.bos.   1/1 
Installed products updated.

Removed:
  katello-ca-consumer-hpe-dl580g7-02.hpe2.lab.eng.bos.redhat.com-1.0-1.noarch   

Complete!
[SUCCESS], [2019-08-23 02:37:53], [/usr/bin/yum -y remove 'katello-ca-consumer-*' katello-agent gofer katello-host-tools katello-host-tools-fact-plugin], completed successfully.
[SUCCESS], [2019-08-23 02:37:53], [Removing /etc/rhsm/ca/katello-server-ca.pem], completed successfully.
[NOTIFICATION], [2019-08-23 02:37:53], [Writing custom cURL configuration to allow download via HTTPS without certificate verification]
[NOTIFICATION], [2019-08-23 02:37:53], [Retrieving Client CA Certificate RPMs]
[RUNNING], [2019-08-23 02:37:53], [rpm -Uvh https://hpe-dl580g7-02.hpe2.lab.eng.bos.redhat.com/pub/katello-ca-consumer-latest.noarch.rpm]
Retrieving https://hpe-dl580g7-02.hpe2.lab.eng.bos.redhat.com/pub/katello-ca-consumer-latest.noarch.rpm
Verifying...                          ########################################
Preparing...                          ########################################
Updating / installing...
katello-ca-consumer-hpe-dl580g7-02.hpe########################################
[SUCCESS], [2019-08-23 02:37:57], [rpm -Uvh https://hpe-dl580g7-02.hpe2.lab.eng.bos.redhat.com/pub/katello-ca-consumer-latest.noarch.rpm], completed successfully.
[NOTIFICATION], [2019-08-23 02:37:57], [Deleting cURL configuration]
[SUCCESS], [2019-08-23 02:37:57], [Removing /tmp/tmp70c6v3ly], completed successfully.
[NOTIFICATION], [2019-08-23 02:37:57], [Writing FQDN katello-fact]
[ERROR], [2019-08-23 02:37:57], EXITING: [0 element in array for search key 'title="hg"' in API '/api/v2/hostgroups'. Please note that all searches are case-sensitive.] failed to execute properly.
[ERROR], [2019-08-23 02:37:57], EXITING: [Please also ensure that the user has permissions to view the searched objects. Fatal error.] failed to execute properly.

Peter,

The first notion of DMI UUID uniqueness was added in Candlepin. The reasoning was for virt-who hypervisor checkins which creates new host records if one does not exist. It was found to create many for the same hypervisors in some cases, and the approach of looking up the DMI UUID was implemented in favor of blindly creating new host profiles.

Satellite/Katello made changes to be compatible with that, and here we are with checking for hostname & DMI UUID uniqeness.

Johathan,

This confirms why i asked the question. And for me it is the argumentation that dmi-uuid should be available directly under the host and not in the dynamic facts array.

Peter

Hi Peter,

I more or less agree with you but is there any negative effect right now by not having it as a proper host field? As I mentioned previously I think some of the value is diminished because we can't make that field unique; there are older installations which have DMI UUID duplicates. There could be slight performance impacts in large installations with many Content Hosts. Feel free to file a Bugzilla!

Jonathon,

With 6.5.2.1 the problem is solved the re-register issue also after build=yes because it now handles the situation of non-existing fact system::dmi::uuid

For me as end user it is working.
From a software-engineer maintenance point of view the current code in the subscription_facet file to do the lookup of the dmi value is complex and prone to errors when a fact changes something changes or when another fact source is added.

Peter

Verified on satellite-6.5.2.1-1.el7sat.noarch, but tfm-rubygem-katello is "3.10.0.55-1" not "3.12.0.12-1"


[root@hpe-dl580g7-02 ~]# rpm -q satellite-6.5.2.1
satellite-6.5.2.1-1.el7sat.noarch

[root@hpe-dl580g7-02 ~]# rpm -q tfm-rubygem-katello
tfm-rubygem-katello-3.10.0.55-1.el7sat.noarch

Verified on Build HEL-8.1.0-20190903.1 Satellite 6.5.2.1, this issue has been fixed.
subscription-manager-1.25.15-1.el8.x86_64
dnf-plugin-subscription-manager-1.25.15-1.el8.x86_64
subscription-manager-cockpit-1.25.15-1.el8.noarch
python3-subscription-manager-rhsm-1.25.15-1.el8.x86_64
subscription-manager-rhsm-certificates-1.25.15-1.el8.x86_64
tfm-rubygem-katello-3.10.0.55-1.el7sat.noarch
katello-3.10.0-0.6.rc1.el7sat.noarch
candlepin-2.5.17-1.el7sat.noarch

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3172