Bug 1486048

Summary:	Activation Key Content Overrides - Unexpected Behavior "No (Default)" repos being enabled by subscription-manager
Product:	[Community] Candlepin	Reporter:	Jonathon Turel <jturel>
Component:	candlepin	Assignee:	candlepin-bugs
Status:	CLOSED WONTFIX	QA Contact:
Severity:	medium	Docs Contact:
Priority:	medium
Version:	2.0	CC:	khowell, nmoumoul, redakkan
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-02-02 12:59:26 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jonathon Turel 2017-08-28 20:31:06 UTC

h1. Summary

I posted on the users mailing list about this a couple of weeks ago, I called it "Activation keys not working as expected" but I think I see what's going on now.

It seems that "No (Default)" is misleading and I can't tell if the problem is subscription-manager, katello or candlepin.

In the end I am seeing that subscription manager retrieves a list of content overrides and a list of available repos from katello(candlepin).  Subscription-manager seems only to reject repos that are explicitly overridden to enabled=0.  So if a repo is available but not overridden it will be added to a host.

This is compounded by the other bug I created #19274  about "Override to no" being missing from the Web UI.

h2. Environment

Katello

<pre>
[alan@katello ~]$ cat /etc/centos-release
CentOS Linux release 7.3.1611 (Core)

[alan@katello ~]$ rpm -q katello foreman candlepin
katello-3.3.1-1.el7.noarch
foreman-1.14.3-1.el7.noarch
candlepin-0.9.54.10-1.el7.noarch
</pre>

Client

<pre>
[root@rhsm-client ~]# cat /etc/centos-release
CentOS Linux release 7.3.1611 (Core)
[root@rhsm-client ~]# rpm -q subscription-manager
subscription-manager-1.17.15-1.el7.centos.x86_64
</pre>

h2. Original State

The goal is to have machines register with the activation key and question and get only "base" repos, centos-os, centos-updates etc.  Other hosts providing different keys may have more repos enabled.  One would think that because "No (Default)" that all I would have to do is enable the repos I want.

h3. Katello Server

<pre>
[alan@katello ~]$ hammer activation-key product-content --id 6 | grep -v -e 'CentOS 6' -e 'centos-6' -e 'el6' -e 'EPEL 6'
--------------|---------------------------------------------------|------|-----|---------|----------------------------------------------------|----------|---------
ID            | NAME                                              | TYPE | URL | GPG KEY | LABEL                                              | ENABLED? | OVERRIDE
--------------|---------------------------------------------------|------|-----|---------|----------------------------------------------------|----------|---------
1490920072788 | Katello 3.3 el7-x86_64 Client                     |      |     |         | My_Org_katello_katello-3_3-el7-x86_64-client       | no       | 1
1490920076693 | Katello 3.3 el7-x86_64                            |      |     |         | My_Org_katello_katello-3_3-el7-x86_64              | no       | default
1490920083144 | Katello 3.3 el7-x86_64 Candlepin                  |      |     |         | My_Org_katello_katello-3_3-el7-x86_64-candlepin    | no       | default
1490920079560 | Katello 3.3 el7-x86_64 Pulp                       |      |     |         | My_Org_katello_katello-3_3-el7-x86_64-pulp         | no       | default
1490920069104 | Puppet Labs Agent 3.7.2 el7 x86_64                |      |     |         | My_Org_puppet-enterprise_pe-3_7_2-el7-x86_64-agent | no       | 1
1490920089132 | Foreman 1.14 el7-x86_64 Plugins                   |      |     |         | My_Org_foreman_foreman-1_14-el7-x86_64-plugins     | no       | default
1490920086679 | Foreman 1.14 el7-x86_64                           |      |     |         | My_Org_foreman_foreman-1_14-el7-x86_64             | no       | default
1490920065429 | EPEL 7 x86_64                                     |      |     |         | My_Org_epel_epel-7-x86_64                          | no       | 1
1490920054709 | CentOS 7 x86_64 Plus                              |      |     |         | My_Org_centos_centos-7-x86_64-centosplus           | no       | default
1490920048716 | CentOS 7 x86_64 Extras                            |      |     |         | My_Org_centos_centos-7-x86_64-extras               | no       | default
1490920040933 | CentOS 7 x86_64 OS                                |      |     |         | My_Org_centos_centos-7-x86_64-os                   | no       | 1
1490920044643 | CentOS 7 x86_64 Updates                           |      |     |         | My_Org_centos_centos-7-x86_64-updates              | no       | 1
1490920058662 | CentOS 7 x86_64 SCLo - sclo                       |      |     |         | My_Org_centos_centos-7-x86_64-sclo-sclo            | no       | default
1490920062617 | CentOS 7 x86_64 SCLo - rh                         |      |     |         | My_Org_centos_centos-7-x86_64-sclo-rh              | no       | default
--------------|---------------------------------------------------|------|-----|---------|----------------------------------------------------|----------|---------
</pre>

h3. Client Machine

<pre>
[root@rhsm-client ~]# subscription-manager register --org="My_Org" --activationkey="ak-centos-7-x86_64-dev" --force
The system with UUID e3cc004f-4df7-4aa7-b86a-480ef5db03c4 has been unregistered
The system has been registered with ID: cab481ec-a610-4806-bfaf-2b71f9891454

No products installed.
[root@rhsm-client ~]# curl https://katello.example.com/rhsm/consumers/cab481ec-a610-4806-bfaf-2b71f9891454/content_overrides --cert /etc/pki/consumer/cert.pem --key /etc/pki/consumer/key.pem
[ {
  "contentLabel" : "My_Org_centos_centos-7-x86_64-os",
  "name" : "enabled",
  "value" : "1",
  "created" : "2017-04-14T01:38:03.824+0000",
  "updated" : "2017-04-14T01:38:03.824+0000"
}, {
  "contentLabel" : "My_Org_epel_epel-7-x86_64",
  "name" : "enabled",
  "value" : "1",
  "created" : "2017-04-14T01:38:03.825+0000",
  "updated" : "2017-04-14T01:38:03.825+0000"
}, {
  "contentLabel" : "My_Org_katello_katello-3_3-el7-x86_64-client",
  "name" : "enabled",
  "value" : "1",
  "created" : "2017-04-14T01:38:03.826+0000",
  "updated" : "2017-04-14T01:38:03.826+0000"
}, {
  "contentLabel" : "My_Org_centos_centos-7-x86_64-updates",
  "name" : "enabled",
  "value" : "1",
  "created" : "2017-04-14T01:38:03.827+0000",
  "updated" : "2017-04-14T01:38:03.827+0000"
}, {
  "contentLabel" : "My_Org_puppet-enterprise_pe-3_7_2-el7-x86_64-agent",
  "name" : "enabled",
  "value" : "1",
  "created" : "2017-04-14T01:38:03.828+0000",
  "updated" : "2017-04-14T01:38:03.828+0000"
} ]
[root@rhsm-client ~subscription-manager repos --list | grep -e ID: -e Enabled:
Repo ID:   My_Org_katello_katello-3_3-el7-x86_64-client
Enabled:   1
Repo ID:   My_Org_katello_katello-3_3-el7-x86_64
Enabled:   1
Repo ID:   My_Org_katello_katello-3_3-el7-x86_64-pulp
Enabled:   1
Repo ID:   My_Org_epel_epel-7-x86_64
Enabled:   1
Repo ID:   My_Org_centos_centos-7-x86_64-sclo-sclo
Enabled:   1
Repo ID:   My_Org_centos_centos-7-x86_64-sclo-rh
Enabled:   1
Repo ID:   My_Org_katello_katello-3_3-el7-x86_64-candlepin
Enabled:   1
Repo ID:   My_Org_puppet-enterprise_pe-3_7_2-el7-x86_64-agent
Enabled:   1
Repo ID:   My_Org_centos_centos-7-x86_64-os
Enabled:   1
Repo ID:   My_Org_centos_centos-7-x86_64-updates
Enabled:   1
Repo ID:   My_Org_centos_centos-7-x86_64-centosplus
Enabled:   1
Repo ID:   My_Org_centos_centos-7-x86_64-extras
Enabled:   1
</pre>

h2. Test Case

If I explicitly override one of the repos to enabled=0 then re-register the client machine we can see that the content override works as expected.

h3. Katello Server

<pre>
[alan@katello ~]$ hammer activation-key content-override --id 6 --content-label My_Org_centos_centos-7-x86_64-extras --value 0
Updated content override

[alan@katello ~]$ hammer activation-key product-content --id 6 | grep -v -e 'CentOS 6' -e 'centos-6' -e 'el6' -e 'EPEL 6'
--------------|---------------------------------------------------|------|-----|---------|----------------------------------------------------|----------|---------
ID            | NAME                                              | TYPE | URL | GPG KEY | LABEL                                              | ENABLED? | OVERRIDE
--------------|---------------------------------------------------|------|-----|---------|----------------------------------------------------|----------|---------
1490920054709 | CentOS 7 x86_64 Plus                              |      |     |         | My_Org_centos_centos-7-x86_64-centosplus           | no       | default
1490920048716 | CentOS 7 x86_64 Extras                            |      |     |         | My_Org_centos_centos-7-x86_64-extras               | no       | 0
1490920040933 | CentOS 7 x86_64 OS                                |      |     |         | My_Org_centos_centos-7-x86_64-os                   | no       | 1
1490920044643 | CentOS 7 x86_64 Updates                           |      |     |         | My_Org_centos_centos-7-x86_64-updates              | no       | 1
1490920058662 | CentOS 7 x86_64 SCLo - sclo                       |      |     |         | My_Org_centos_centos-7-x86_64-sclo-sclo            | no       | default
1490920062617 | CentOS 7 x86_64 SCLo - rh                         |      |     |         | My_Org_centos_centos-7-x86_64-sclo-rh              | no       | default
1490920065429 | EPEL 7 x86_64                                     |      |     |         | My_Org_epel_epel-7-x86_64                          | no       | 1
1490920069104 | Puppet Labs Agent 3.7.2 el7 x86_64                |      |     |         | My_Org_puppet-enterprise_pe-3_7_2-el7-x86_64-agent | no       | 1
1490920072788 | Katello 3.3 el7-x86_64 Client                     |      |     |         | My_Org_katello_katello-3_3-el7-x86_64-client       | no       | 1
1490920076693 | Katello 3.3 el7-x86_64                            |      |     |         | My_Org_katello_katello-3_3-el7-x86_64              | no       | default
1490920083144 | Katello 3.3 el7-x86_64 Candlepin                  |      |     |         | My_Org_katello_katello-3_3-el7-x86_64-candlepin    | no       | default
1490920079560 | Katello 3.3 el7-x86_64 Pulp                       |      |     |         | My_Org_katello_katello-3_3-el7-x86_64-pulp         | no       | default
1490920089132 | Foreman 1.14 el7-x86_64 Plugins                   |      |     |         | My_Org_foreman_foreman-1_14-el7-x86_64-plugins     | no       | default
1490920086679 | Foreman 1.14 el7-x86_64                           |      |     |         | My_Org_foreman_foreman-1_14-el7-x86_64             | no       | default
--------------|---------------------------------------------------|------|-----|---------|----------------------------------------------------|----------|---------
</pre>

h3. Client Machine

<pre>
[root@rhsm-client ~]# subscription-manager register --org="My_Org" --activationkey="ak-centos-7-x86_64-dev" --force                                          
The system with UUID cab481ec-a610-4806-bfaf-2b71f9891454 has been unregistered
The system has been registered with ID: 95d7b160-fb05-4086-8e28-e1b379ac0ab4

No products installed.
[root@rhsm-client ~]# curl https://katello.example.com/rhsm/consumers/95d7b160-fb05-4086-8e28-e1b379ac0ab4/content_overrides --cert /etc/pki/consumer/cert.pem --key /etc/pki/consumer/key.pem
[ {
  "contentLabel" : "My_Org_katello_katello-3_3-el7-x86_64-client",
  "name" : "enabled",
  "value" : "1",
  "created" : "2017-04-14T01:40:56.147+0000",
  "updated" : "2017-04-14T01:40:56.147+0000"
}, {
  "contentLabel" : "My_Org_epel_epel-7-x86_64",
  "name" : "enabled",
  "value" : "1",
  "created" : "2017-04-14T01:40:56.147+0000",
  "updated" : "2017-04-14T01:40:56.147+0000"
}, {
  "contentLabel" : "My_Org_centos_centos-7-x86_64-extras",
  "name" : "enabled",
  "value" : "0",
  "created" : "2017-04-14T01:40:56.148+0000",
  "updated" : "2017-04-14T01:40:56.148+0000"
}, {
  "contentLabel" : "My_Org_centos_centos-7-x86_64-os",
  "name" : "enabled",
  "value" : "1",
  "created" : "2017-04-14T01:40:56.149+0000",
  "updated" : "2017-04-14T01:40:56.149+0000"
}, {
  "contentLabel" : "My_Org_puppet-enterprise_pe-3_7_2-el7-x86_64-agent",
  "name" : "enabled",
  "value" : "1",
  "created" : "2017-04-14T01:40:56.150+0000",
  "updated" : "2017-04-14T01:40:56.150+0000"
}, {
  "contentLabel" : "My_Org_centos_centos-7-x86_64-updates",
  "name" : "enabled",
  "value" : "1",
  "created" : "2017-04-14T01:40:56.151+0000",
  "updated" : "2017-04-14T01:40:56.151+0000"
} ]
[root@rhsm-client ~subscription-manager repos --list | grep -e ID: -e Enabled:                                                                           
Repo ID:   My_Org_katello_katello-3_3-el7-x86_64-client
Enabled:   1
Repo ID:   My_Org_katello_katello-3_3-el7-x86_64
Enabled:   1
Repo ID:   My_Org_katello_katello-3_3-el7-x86_64-pulp
Enabled:   1
Repo ID:   My_Org_epel_epel-7-x86_64
Enabled:   1
Repo ID:   My_Org_centos_centos-7-x86_64-sclo-sclo
Enabled:   1
Repo ID:   My_Org_centos_centos-7-x86_64-sclo-rh
Enabled:   1
Repo ID:   My_Org_katello_katello-3_3-el7-x86_64-candlepin
Enabled:   1
Repo ID:   My_Org_puppet-enterprise_pe-3_7_2-el7-x86_64-agent
Enabled:   1
Repo ID:   My_Org_centos_centos-7-x86_64-os
Enabled:   1
Repo ID:   My_Org_centos_centos-7-x86_64-updates
Enabled:   1
Repo ID:   My_Org_centos_centos-7-x86_64-centosplus
Enabled:   1
Repo ID:   My_Org_centos_centos-7-x86_64-extras
Enabled:   0
</pre>

h1. Speculation

I think what is happening here is that katello(candlepin) are treating absent overrides as disabled.  Where subscription-manager is treating absent overrides as enabled.

So it is hard for me to say if the bug is with katello(candlepin) or subscription-manager.

I think I saw in another ticket that if repo==custom then enabled=default=0 and if repo==redhat then enabled=default=1.  Did I get that correct?  Unfortunately that probably complicates things a bit.

Here's what I can think of to fix this.

# Change the behavior of subscription-manager --> if not explicitly enabled=1 then enabled=0 (this would probably require katello(candlepin) changes anyway because of the redhat|custom behaviorial differences)
# Have katello explicitly enabled=0|1 on activation key creation and based on redhat|custom type
# Have katello(candlepin) explicitly return enabled=0|1 for repos that are enabled='default' based on redhat|custom type

-Alan

Comment 1 Jonathon Turel 2017-08-28 20:31:10 UTC

Created from redmine issue http://projects.theforeman.org/issues/19275

Comment 2 Jonathon Turel 2017-08-28 20:31:12 UTC

Upstream bug assigned to None

Comment 3 Jonathon Turel 2017-08-28 20:35:16 UTC

Not sure how these repos are becoming enabled. Here's some output from a local test:

hammer activation-key product-content --id=1 --organization-id=1
--------------|---------------------------------------------------|------|-----|---------|------------------------------|----------|---------
ID            | NAME                                              | TYPE | URL | GPG KEY | LABEL                        | ENABLED? | OVERRIDE
--------------|---------------------------------------------------|------|-----|---------|------------------------------|----------|---------
2456          | Red Hat Enterprise Linux 7 Server (RPMs)          |      |     |         | rhel-7-server-rpms           | yes      |         
3030          | Red Hat Enterprise Linux 7 Server - Extras (RPMs) |      |     |         | rhel-7-server-extras-rpms    | no       |         
1502203385603 | foo                                               |      |     |         | Default_Organization_foo_foo | yes      |         
--------------|---------------------------------------------------|------|-----|---------|------------------------------|----------|---------


[root@localhost ~]# subscription-manager register --activationkey=testkey --org=Default_Organization --force
The system has been registered with ID: 0f71c9e7-0611-4b51-aca7-546994ce7685 

Installed Product Current Status:
Product Name: Red Hat Enterprise Linux Server
Status:       Subscribed

[root@localhost ~]# subscription-manager repos --list | grep -e ID: -e Enabled:
Repo ID:   rhel-7-server-rpms
Enabled:   1
Repo ID:   rhel-7-server-extras-rpms
Enabled:   1
Repo ID:   Default_Organization_foo_foo
Enabled:   1


No content overrides on the host.

Let me know if I can provide any other details.

Comment 4 Kevin Howell 2017-08-31 14:37:46 UTC

Note: likely not a candlepin problem, but we're leaving it under that component until we know more. Needs investigation.

Comment 7 Nikos Moumoulidis 2023-02-02 12:59:26 UTC

Thank you for your interest in Candlepin. We have evaluated this request, and while we recognize that it is a valid request, we do not expect this to be implemented in the product in the foreseeable future. This is due to other priorities for the project, and not a reflection on the request itself. We are therefore closing this out as WONTFIX. If you have any concerns about this feel free to open a new bug with more up to date details. Thank you.