Created attachment 1059573 [details] foreman-debug from server failing to show full list of packages for repo Description of problem: I had a failure downloading 1 package in Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server. This repo contains almost 16,000 packages, which downloaded successfully save the one, however when I did I yum repolist on a client I saw: [root@mac52540066f981 ~]# yum repolist Loaded plugins: package_upload, product-id, security, subscription-manager jb-eap-6-for-rhel-6-server-rpms | 2.1 kB 00:00 rhel-6-server-rhev-mgmt-agent-rpms | 2.1 kB 00:00 rhel-6-server-rhevm-3.5-rpms | 2.1 kB 00:00 rhel-6-server-rpms | 2.1 kB 00:00 rhel-6-server-satellite-tools-6-beta-rpms | 2.1 kB 00:00 rhel-6-server-supplementary-rpms | 2.1 kB 00:00 repo id repo name status jb-eap-6-for-rhel-6-server-rpms JBoss Enterprise Application Platform 6 (RHEL 6 Server) (RPMs) 2,340 rhel-6-server-rhev-mgmt-agent-rpms Red Hat Enterprise Virtualization Management Agents (RPMs) 538 rhel-6-server-rhevm-3.5-rpms Red Hat Enterprise Virtualization Manager 3.5 (RPMs) 188 rhel-6-server-rpms Red Hat Enterprise Linux 6 Server (RPMs) 0 rhel-6-server-satellite-tools-6-beta-rpms Red Hat Satellite Tools 6 Beta (for RHEL 6 Server) (RPMs) 53 rhel-6-server-supplementary-rpms Red Hat Enterprise Linux 6 Server - Supplementary (RPMs) 537 repolist: 3,656 After sync'ing the repo and picking up the one package that failed to download I have: [root@mac52540066f981 ~]# yum repolist Loaded plugins: package_upload, product-id, security, subscription-manager jb-eap-6-for-rhel-6-server-rpms | 2.1 kB 00:00 rhel-6-server-rhev-mgmt-agent-rpms | 2.1 kB 00:00 rhel-6-server-rhevm-3.5-rpms | 2.1 kB 00:00 rhel-6-server-rpms | 2.1 kB 00:00 rhel-6-server-satellite-tools-6-beta-rpms | 2.1 kB 00:00 rhel-6-server-supplementary-rpms | 2.1 kB 00:00 repo id repo name status jb-eap-6-for-rhel-6-server-rpms JBoss Enterprise Application Platform 6 (RHEL 6 Server) (RPMs) 2,340 rhel-6-server-rhev-mgmt-agent-rpms Red Hat Enterprise Virtualization Management Agents (RPMs) 538 rhel-6-server-rhevm-3.5-rpms Red Hat Enterprise Virtualization Manager 3.5 (RPMs) 188 rhel-6-server-rpms Red Hat Enterprise Linux 6 Server (RPMs) 1 rhel-6-server-satellite-tools-6-beta-rpms Red Hat Satellite Tools 6 Beta (for RHEL 6 Server) (RPMs) 53 rhel-6-server-supplementary-rpms Red Hat Enterprise Linux 6 Server - Supplementary (RPMs) 537 repolist: 3,657 When I try to download dependencies that should be available from that repo they are not. Version-Release number of selected component (if applicable): pulp-server-2.6.0.15-1.el7sat.noarch How reproducible: Always, in so far as no matter how many times I resync the repo I have jus tone package available. Steps to Reproduce: 1.Try to sync a large repo 2.Somehow simulate a failure downloading a package Actual results: It seems none of the successfully downloaded packages ever become available. Expected results: I should be able to use the download packages; upon sync'ing missing packages everything should be available. Additional info: foreman-debug from server attached.
Looking at the logs, a celery process was OOM-killed while doing a publish. That process was not using an unreasonable amount of RAM, and it was killed immediately after some big java processes started. The system only had 8GB of ram, but we apparently require 12. The fact that the publish was killed triggered this bug: https://pulp.plan.io/issues/1284 The "last_published" timestamp was updated before the publish had succeeded. The next publish was "incremental", and only added new rpms to the existing metadata (which was empty). This is why the published repo appeared to have only 1 rpm. This was fixed in pulp 2.7.0. A work-around would be to either manually change that timestamp value, or remove any unit from the repo. Either of those will force the next publish to be a full one instead of incremental.
Based on comment #3 and discussion with Jason, we are going to close this bug out. Since the target systems had RAM increased to the Satellite 6 minimum recommendation, the issue has no longer been observed.