Description of problem: On-demand lazy sync in a load-balanced content capsule environment yields 403 forbidden errors. If the capsules are set up in a load-balanced fashion, and the the repos are set to sync on-demand, then a client's yum attempt can fail for some subset of the requested packages. A subsequent attempt load-balancing to a different capsule will fail for a different subset of packages. In the /var/log/httpd/pulp-https_access_ssl.log for the capsule, the failed attempts are all attempting "GET /streamer/..." while the successful pulls are from "GET /pulp/repos/..." (we believe this implies the content does not exist on the capsule and so must be requested on-demand). Version-Release number of selected component (if applicable): Satellite v6.2.6 How reproducible: Steps to Reproduce: 1. Set up a load balanced environment: https://access.redhat.com/articles/2474581 2. Set Repo syncing as "on-demand" and sync all capsules. 3. From clients, attempt to "yum install" a number of packages that have never previously been installed through the capsules before. Actual results: Some RPMs may fail with a "403 Forbidden". In the capsule' /var/log/httpd/pulp-https_access_ssl.log log, for each 403 Forbidden attempt, there is a corresponding "GET /streamer/..." Expected results: The yum attempt is able to pull all the RPMs successfully, even if they have to pull via /streamer/ ... Additional info: Changing sync policy to "immediate" and re-syncing capsules got rid of the "403 forbidden" errors. The load-balancing is thought to be involved, in that if it directs to a different capsule, different packages trigger the 403 forbidden. A stickiness of 30 minutes was set to keep subsequent requests hitting the same capsule for the short term, but this does not help with the missing lazy content.
I think I know why this is happening. When pulp fails to find a file stored locally, it redirects the client to a signed URL that gets handled by a different service (we call it the "streamer"). That service verifies the signature, and then it proceeds with retrieving the file on-demand. The shared secret is the RSA key pair referenced in /etc/pulp/server.conf under the [authentication] section. That RSA key pair gets created at install time, so it should be different on each pulp installation. If you want multiple pulp installations to trust each other's redirects, they need to all be configured with the same RSA key pair. You can read all about the workflow here for more details: http://docs.pulpproject.org/dev-guide/design/deferred-download.html Jeff, am I missing anything? If they change the key pair and restart pulp services, should this work?
Mike, nope. That should work just fine. It's worth noting that fix bug:1346096 has a PR for the upstream issue that moves RSA key pair generation from the RPM %post to a manual post-install step using a new pulp-generate-key-pair script. This wont really help in this case but does draw attention to managing the key pair. https://bugzilla.redhat.com/show_bug.cgi?id=1346096
Great. Thanks Jeff for the confirmation. Since the problem can be fixed with a configuration change, I'm closing this as not a bug. Please let us know if you have any additional questions.
We are hitting the same issue on Satellite 6.4 using the load balancing guide. Is there any additional configuration needed? Do the rsa key in /etc/pulp/server.conf need to be the same on all the capsule? Cyril
No additional configuration is needed. Yes, the RSA key pair needs to be the same on all the capsules behind the load balancer to support redirect between them.
Removing hte 6.2.z flag and adding backlog.
The Satellite Team is attempting to provide an accurate backlog of bugzilla requests which we feel will be resolved in the next few releases. We do not believe this bugzilla will meet that criteria, and have plans to close it out in 1 month. This is not a reflection on the validity of the request, but a reflection of the many priorities for the product. If you have any concerns about this, feel free to contact Rich Jerrido or Bryan Kearney or your account team. If we do not hear from you, we will close this bug out. Thank you.
Thank you for your interest in Satellite 6. We have evaluated this request, and while we recognize that it is a valid request, we do not expect this to be implemented in the product in the foreseeable future. This is due to other priorities for the product, and not a reflection on the request itself. We are therefore closing this out as WONTFIX. If you have any concerns about this, please do not reopen. Instead, feel free to contact Red Hat Technical Support. Thank you.