Bug 1264964 - subscription-manager package profile submission is sending profiles with UUID=None to SLE endpoint
subscription-manager package profile submission is sending profiles with UUID...
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: subscription-manager (Show other bugs)
Unspecified Unspecified
medium Severity high
: rc
: ---
Assigned To: Adrian Likins
John Sefler
: Reopened, Triaged
Depends On:
  Show dependency treegraph
Reported: 2015-09-21 14:14 EDT by venkat
Modified: 2016-11-03 16:27 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1332323 (view as bug list)
Last Closed: 2016-11-03 16:27:17 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:2592 normal SHIPPED_LIVE Moderate: subscription-manager security, bug fix, and enhancement update 2016-11-03 08:10:42 EDT

  None (edit)
Description venkat 2015-09-21 14:14:41 EDT
Description of problem:

subscription-manager package profile submission is sending profiles with UUID=None to SLE endpoint

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:

Additional info:
Comment 2 Shayne Riley 2015-09-22 16:16:03 EDT
This was originally discovered September 9 when looking more in-depth into our Splunk logs for C3. I noticed that we're getting a lot of these calls to C3 to upload profiles:

PUT /errata-cron/rest/package-profile/None

What we normally expect instead is a UUID in place of  "None", like this:

PUT /errata-cron/rest/package-profile/000138d2-0ef0-404c-947a-14c1d78f8935.

These "PUT .../None" calls aren't a significant portion of our "upload profile" traffic; On Tuesday, we received 18,475 profiles, and 384 of those profiles were to "None", which is just over 2% of all the traffic we receive.

Looking back through Splunk, it was discovered that this behavior occurred in C3 as far back as July 1, when we first started advertising the URL to 1% of the clients. (I had originally thought that we started seeing these after our August 28 release which included C3's fix for a small set of traffic which did not have the correct "CP-consumer" header when they attempted to upload their profiles. This turned out not to be the case and we were receiving the uuid=None profiles from the very beginning).

With more investigation using splunk, we determined that calls regarding uuid=None were also being made to Candlepin.

One of the theories was that this could be caused with certain user behavior in subscription-manager-gui, but this was abandoned when it was discovered that many of the systems with uuid=None didn't have the gui installed anyway.

Where we stand right now is that we can clearly see when calls to Candlepin and C3 with uuid=None occur. However we have yet to establish the cause of this.
Comment 4 Shayne Riley 2015-09-22 17:28:10 EDT
From Adrian Likins email on Sept 11:

Attempted to reproduce the scenarios I thought were most likely to cause
those requests, but had no luck.

And after exploring the logs in Splunk, I think the source is something semi-automated,
so unlikely gui related. That most likely points to rhsmcertd.

Some notes from the info about the requests in splunk:

- over the last week, there are about:
- 2600 requests to consumers/None/packages
- 5200 requests to GET consumer/None/release

- Most of those are from the top dozen or so ip addresses
- The top ip addresses are making those request 1 time a minute
  (pointing to daemon/script/automation and not gui)
- Some of the top ip addresses are internal machines. Including some that
  are used for errata testing. I've contacted the machine owners to see if I
  can get logs/configs/shell for troubleshooting.
- Some (~10%) of the GET /consumers/None/release are invoked from the subscription-manager plugin in yum
  - (and only ~rhel7.2ish machines would be reporting the originating command in the user agent, so not representative)

For the package profiles for consumer=None that have been collected, I'd like to see the info about
the count and versions of systems with following packages:
  - subscription-manager
  - subscription-manager-gui
  - virt-who
  - python-rhsm
  - yum
  - python
  - python-simplejson
  - m2crypto
  - rhn-client-tools

I'm hoping I can get access to one of the internal machines that are making
these requests soon.
Comment 7 Adrian Likins 2015-09-23 12:37:09 EDT
A few other splunk queries:

# check candlepin logs (access and log4j) for /None requests that make it to prod
(This should also extract the request id ('req=') that can be
followed for a little more detail. )

host="s*.candlepin.prod.*"  index=rh_jboss "consumers/None"

Splunk note: Last week the splunk timestamps for the requests to F5 had were wrong, often by a lot (but also my non-hour increments, so not just timezones)
If you see a F5 request in splunk, double check the splunk chose timestamp against the timestamp in the log entry, they are often different.
Comment 13 John Sefler 2015-10-02 10:08:21 EDT
deferring to rhel-7.3.0 due to lack of a reproducer and time schedule remaining for remaining rhel-7.2.0
Comment 22 Adrian Likins 2015-10-13 18:17:50 EDT
has a subman workaround for this.
Comment 27 John Sefler 2016-09-28 11:54:57 EDT
This is a difficult bug to explicitly demonstrate through functional use of an offending version of subscription-manager that will cause it to issue a PUT request to /subscription/consumers/None/packages.  However, I can see in the files changed of the pull request from  comment 22 that a new developer unit test_update_check_consumer_uuid_none has been written for coverage of this bug.  Since the latest jenkins pull request build shows...



test.test_cache.TestProfileManager.test_update_check_consumer_uuid_none (from nosetests)

... I am moving this bug to VERIFIED.
Comment 29 errata-xmlrpc 2016-11-03 16:27:17 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.