Bug 1264964

This was originally discovered September 9 when looking more in-depth into our Splunk logs for C3. I noticed that we're getting a lot of these calls to C3 to upload profiles:

PUT /errata-cron/rest/package-profile/None

What we normally expect instead is a UUID in place of  "None", like this:

PUT /errata-cron/rest/package-profile/000138d2-0ef0-404c-947a-14c1d78f8935.

These "PUT .../None" calls aren't a significant portion of our "upload profile" traffic; On Tuesday, we received 18,475 profiles, and 384 of those profiles were to "None", which is just over 2% of all the traffic we receive.

Looking back through Splunk, it was discovered that this behavior occurred in C3 as far back as July 1, when we first started advertising the URL to 1% of the clients. (I had originally thought that we started seeing these after our August 28 release which included C3's fix for a small set of traffic which did not have the correct "CP-consumer" header when they attempted to upload their profiles. This turned out not to be the case and we were receiving the uuid=None profiles from the very beginning).

With more investigation using splunk, we determined that calls regarding uuid=None were also being made to Candlepin.

One of the theories was that this could be caused with certain user behavior in subscription-manager-gui, but this was abandoned when it was discovered that many of the systems with uuid=None didn't have the gui installed anyway.

Where we stand right now is that we can clearly see when calls to Candlepin and C3 with uuid=None occur. However we have yet to establish the cause of this.

From Adrian Likins email on Sept 11:

Attempted to reproduce the scenarios I thought were most likely to cause
those requests, but had no luck.

And after exploring the logs in Splunk, I think the source is something semi-automated,
so unlikely gui related. That most likely points to rhsmcertd.

Some notes from the info about the requests in splunk:

- over the last week, there are about:
- 2600 requests to consumers/None/packages
- 5200 requests to GET consumer/None/release

- Most of those are from the top dozen or so ip addresses
- The top ip addresses are making those request 1 time a minute
  (pointing to daemon/script/automation and not gui)
- Some of the top ip addresses are internal machines. Including some that
  are used for errata testing. I've contacted the machine owners to see if I
  can get logs/configs/shell for troubleshooting.
- Some (~10%) of the GET /consumers/None/release are invoked from the subscription-manager plugin in yum
  - (and only ~rhel7.2ish machines would be reporting the originating command in the user agent, so not representative)


For the package profiles for consumer=None that have been collected, I'd like to see the info about
the count and versions of systems with following packages:
  - subscription-manager
  - subscription-manager-gui
  - virt-who
  - python-rhsm
  - yum
  - python
  - python-simplejson
  - m2crypto
  - rhn-client-tools

I'm hoping I can get access to one of the internal machines that are making
these requests soon.

A few other splunk queries:

# check candlepin logs (access and log4j) for /None requests that make it to prod
(This should also extract the request id ('req=') that can be
followed for a little more detail. )

host="s*.candlepin.prod.*"  index=rh_jboss "consumers/None"


Splunk note: Last week the splunk timestamps for the requests to F5 had were wrong, often by a lot (but also my non-hour increments, so not just timezones)
If you see a F5 request in splunk, double check the splunk chose timestamp against the timestamp in the log entry, they are often different.

deferring to rhel-7.3.0 due to lack of a reproducer and time schedule remaining for remaining rhel-7.2.0

https://github.com/candlepin/subscription-manager/pull/1326
has a subman workaround for this.

This is a difficult bug to explicitly demonstrate through functional use of an offending version of subscription-manager that will cause it to issue a PUT request to /subscription/consumers/None/packages.  However, I can see in the files changed of the pull request from  comment 22 that a new developer unit test_update_check_consumer_uuid_none has been written for coverage of this bug.  Since the latest jenkins pull request build shows...

https://rhsm-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/subscription-manager-nose-tests-pull-request-builder/lastSuccessfulBuild/testReport/test.test_cache/TestProfileManager/test_update_check_consumer_uuid_none/

Passed

test.test_cache.TestProfileManager.test_update_check_consumer_uuid_none (from nosetests)

... I am moving this bug to VERIFIED.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2592.html