Description of problem: During a reposync from the cdn Satellite 6 fails to pull a populated version of the *-updateinfo.xml.gz file from the CDN. This file is responsible for operation of the --bugfix and --security options of yum. Version-Release number of selected component (if applicable): How reproducible: Unknown. Steps to Reproduce: 1. Build a satellite 6.2.2 2. Sync with the CDN 3. On a registered client, try yum update --bugfix (or any other combination of this flag) Actual results: No packages needed for security; 237 packages available" Expected results: [root@demo ~]# yum update --downloadonly --bugfix --downloaddir=/tmp Loaded plugins: package_upload, product-id, search-disabled-repos, subscription-manager rhel-7-server-rpms | 2.0 kB 00:00:00 rhel-7-server-satellite-tools-6.2-rpms | 2.1 kB 00:00:00 rhel-server-rhscl-7-rpms | 2.0 kB 00:00:00 --> openssh-server-6.6.1p1-25.el7_2.x86_64 from rhel-7-server-rpms removed (updateinfo) ------------------------- . . . ------------------------- ---> Package util-linux.x86_64 0:2.23.2-26.el7_2.3 will be an update --> Finished Dependency Resolution Dependencies Resolved =============================================================================================== Package Arch Version Repository Size =============================================================================================== Installing: kernel x86_64 3.10.0-327.28.3.el7 rhel-7-server-rpms 33 M Updating: NetworkManager x86_64 1:1.0.6-30.el7_2 rhel-7-server-rpms 2.0 M ------------- . . . ------------- util-linux x86_64 2.23.2-26.el7_2.3 rhel-7-server-rpms 1.9 M Transaction Summary =============================================================================================== Install 1 Package Upgrade 61 Packages Total download size: 89 M Background downloading packages, then exiting: Delta RPMs disabled because /usr/bin/applydeltarpm not installed. (1/62): NetworkManager-config-server-1.0.6-30.el7_2.x86_64.rpm | 122 kB 00:00:01 ------------------------ . . . ------------------------ (62/62): util-linux-2.23.2-26.el7_2.3.x86_64.rpm | 1.9 MB 00:00:00 ----------------------------------------------------------------------------------------------- Total 2.0 MB/s | 89 MB 00:00:43 exiting because "Download Only" specified [root@demo ~]# Additional info:
I'm adjusting the title to focus on the behavior instead of an early diagnosis. Feel free to tweak or revise it.
I've associated an upstream bug that I strongly suspect is the cause. Looking at the reproducer system, the repo contents look correct, and the most recent publish happened after there were errata associated with the repo. None the less, the updateinfo file referenced in repomd.xml has no erratum. Old updateinfo files are preserved in published repos, even when publish tasks create new ones. In this case, I saw multiple updateinfo files, most of them empty. The upstream bug could be triggered if the mtime of one of the older files was modified to make it appear as the newest. Justin, is it possible that katello's clone_distributor is causing those mtimes to change?
Adding the evidence from the satellite server's published data. repomd.xml references updateinfo file "6721a84420075fbb3b48944dd7c70d45b43644a0-updateinfo.xml.gz" The "correct" updateinfo file, based on the one that is actually populated, appears to be e2b23b61ea1b24c7c53fc73e87eb637e89753485-updateinfo.xml.gz You can see below that the one referenced by repomd.xml does have a newer mtime than the correct one. [root@sat6 repodata]# stat 6721a84420075fbb3b48944dd7c70d45b43644a0-updateinfo.xml.gz File: ‘6721a84420075fbb3b48944dd7c70d45b43644a0-updateinfo.xml.gz’ Size: 93 Blocks: 8 IO Block: 4096 regular file Device: fd03h/64771d Inode: 537178272 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 48/ apache) Gid: ( 48/ apache) Context: system_u:object_r:httpd_sys_rw_content_t:s0 Access: 2016-10-12 12:57:08.849982303 -0400 Modify: 2016-10-12 12:55:04.314902263 -0400 Change: 2016-10-12 12:55:04.314902263 -0400 Birth: - [root@sat6 repodata]# stat e2b23b61ea1b24c7c53fc73e87eb637e89753485-updateinfo.xml.gz File: ‘e2b23b61ea1b24c7c53fc73e87eb637e89753485-updateinfo.xml.gz’ Size: 1372039 Blocks: 2680 IO Block: 4096 regular file Device: fd03h/64771d Inode: 577090611 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 48/ apache) Gid: ( 48/ apache) Context: system_u:object_r:httpd_sys_rw_content_t:s0 Access: 2016-10-12 16:09:42.644299334 -0400 Modify: 2016-10-12 12:55:04.292902249 -0400 Change: 2016-10-12 12:55:04.292902249 -0400 Birth: -
According to an in-person chat with Justin, since this is a Library repo, the clone distributor would not have been used. The only ready explanation then is that some other actor moved, copied, or modified the files such that the mtimes of the various updateinfo files no longer reflected the order in which they were originally created. As far as we know, when pulp is the only actor managing those files, the mtimes should be preserved and this bug will not be triggered. To work around the problem, you can delete the published directory and then re-publish the repo. The path to delete would be: /var/lib/pulp/published/yum/master/yum_distributor/<repo_id>/<timestamp>
The workaround does work for me. Am I doing something wrong? I deleted such a repository: /var/lib/pulp/published/yum/master/yum_distributor/<ORG>-<REPO>/<TIMESTAMP> Content-view publication failed with: 184: Actions::Pulp::Repository::DistributorPublish (error) [ 0.67s / 0.15s ] Source Directory (/var/lib/pulp/published/yum/master/yum_distributor/<ORG>-<REPO>/<TIMESTAMP>), does not exist, cannot publish.
Pierre, That error appears to come from katello's "clone distributor". Specifically here: https://github.com/Katello/pulp-katello/blob/master/pulp_katello/distributors/yum_clone_distributor/distributor.py#L186 Justin, any advice? At least when it's just pulp doing the publish, if it notices that the directory is not present, it happily does a full publish. Is publication with pulp's yum_distributor somehow being skipped?
The following workaround worked for me. For each repository directory in /var/lib/pulp/published/yum/master/yum_distributor/<ORGANIZATION>-{Red_Hat_,Oracle_,JBoss_}*, if its repodata/repond.xml file references an empty updateinfo.xml.gz file, then delete the directory <repo_path>/<timestamp>. Afterward, in the satellite webUI, perform a synchronization of the corresponding products.
I think when you said to do a publish, you meant a yum repo publish, whereas Pierre took that to mean a content view publish. In this case yes, its trying to copy metadata that isn't there. Pierre, We need to do a repository publish. To do that, find the repository ID of the repo in question, navigate to Content > Products > click your product > click your repo and then look in the URL bar, you should see something like this: /products/7/repositories/55 take that 2nd numerical id (in this case 55) and then run the following (replacing 55 with your repo id): foreman-rake console User.current = User.anonymous_api_admin repo = Katello::Repository.find(55) ForemanTasks.sync_task(Actions::Katello::Repository::MetadataGenerate, repo) Then you can resume the content view publish and it should complete.
Pierre, yep, resyncing also would have accomplished this as well.
So I ran through the steps above and got some 'interesting' results: When I ran the MetadataGenerate I caught it creating a timestampped location of 1476366777.73 then after a few minutes, it created a new one 1476367202.53 The new one, references the invalid updateinfo.xml.gz Don't know if this helps or not... irb(main):004:0* repo=Katello::Repository.find(1) => #<Katello::Repository id: 1, name: "Red Hat Enterprise Linux 6 Server RPMs x86_64 6Ser...", pulp_id: "<org name>-Red_Hat_Enterprise_Lin...", created_at: "2016-08-08 17:40:07", updated_at: "2016-08-08 20:04:47", major: 6, minor: "6Server", gpg_key_id: nil, cp_label: "rhel-6-server-rpms", library_instance_id: nil, content_id: "168", arch: "x86_64", label: "Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Ser...", content_view_version_id: 1, relative_path: "<org name>/Library/content/dist/r...", url: "https://cdn.redhat.com/content/dist/rhel/server/6/...", unprotected: false, content_type: "yum", product_id: 8, environment_id: 1, checksum_type: "sha1", docker_upstream_name: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, distribution_uuid: nil, mirror_on_sync: true, download_policy: "immediate"> irb(main):005:0> ForemanTasks.sync_task(Actions::Katello::Repository::MetadataGenerate, repo) => #<ForemanTasks::Task::DynflowTask id: "26c1c818-7ac3-4e23-b32e-8434b38c93c6", type: "ForemanTasks::Task::DynflowTask", label: "Actions::Katello::Repository::MetadataGenerate", started_at: "2016-10-13 13:55:38", ended_at: "2016-10-13 14:01:25", state: "stopped", result: "success", external_id: "1d0cecbe-6e7e-4421-8486-c30e4b7907d9", parent_task_id: nil, start_at: "2016-10-13 13:55:38", start_before: nil> [root@ 1476366777.73]# ls -la repodata/ total 82140 drwxr-xr-x. 2 apache apache 4096 Oct 13 09:59 . drwxr-xr-x. 3 apache apache 966656 Oct 13 10:00 .. -rw-r--r--. 1 apache apache 1101933 Oct 13 09:59 54e59ffa906d073e1df27b4bed987dd7252e5bc9-comps.xml -rw-r--r--. 1 apache apache 3676462 Oct 13 09:59 9d14513866957c4cb1f9d6cb3f04b231be5b0876-updateinfo.xml.gz -rw-r--r--. 1 apache apache 29411929 Oct 13 09:59 bf9808c80fc09366be23ecd535070a4959688874-primary.xml.gz -rw-r--r--. 1 apache apache 34443697 Oct 13 09:59 c0a3bac677383998645d012ac799beec6095c886-filelists.xml.gz -rw-r--r--. 1 apache apache 14231598 Oct 13 09:59 d6d089072215531e7b970db5134ceeee79774573-other.xml.gz lrwxrwxrwx. 1 apache apache 126 Oct 13 09:59 productid -> /var/lib/pulp/content/units/yum_repo_metadata_file/04/acbfa3dff8929de3fff11443183879e710ea7c1896d65bc728c270fae6f874/productid root@sat 1476367202.53]# cd repodata/ [root@sat repodata]# ls -la total 82140 drwxr-xr-x. 2 apache apache 4096 Oct 13 10:01 . drwxr-xr-x. 3 apache apache 966656 Oct 13 10:01 .. -rw-r--r--. 1 apache apache 14231598 Oct 13 10:01 0340275b6f931489c8663f6ea51b71d7e3eefc75-other.xml.gz -rw-r--r--. 1 apache apache 93 Oct 13 10:01 0d11569ff8d032e17c471b13fe426ffc013c0039-updateinfo.xml.gz -rw-r--r--. 1 apache apache 1101933 Oct 13 10:01 54e59ffa906d073e1df27b4bed987dd7252e5bc9-comps.xml -rw-r--r--. 1 apache apache 29411929 Oct 13 10:01 696ec2025734b26832dc980280c5390a494fbaa7-primary.xml.gz -rw-r--r--. 1 apache apache 34443697 Oct 13 10:01 7a42eba03c74776be72deba4c94e2f0e9bdcacd5-filelists.xml.gz -rw-r--r--. 1 apache apache 3676462 Oct 13 10:01 9d14513866957c4cb1f9d6cb3f04b231be5b0876-updateinfo.xml.gz lrwxrwxrwx. 1 apache apache 126 Oct 13 10:01 productid -> /var/lib/pulp/content/units/yum_repo_metadata_file/04/acbfa3dff8929de3fff11443183879e710ea7c1896d65bc728c270fae6f874/productid [root@sat repodata]# ls -ltr total 80948 -rw-r--r--. 1 apache apache 2016 Oct 13 10:01 repomd.xml -rw-r--r--. 1 apache apache 3676462 Oct 13 10:01 9d14513866957c4cb1f9d6cb3f04b231be5b0876-updateinfo.xml.gz lrwxrwxrwx. 1 apache apache 126 Oct 13 10:01 productid -> /var/lib/pulp/content/units/yum_repo_metadata_file/04/acbfa3dff8929de3fff11443183879e710ea7c1896d65bc728c270fae6f874/productid -rw-r--r--. 1 apache apache 1101933 Oct 13 10:01 54e59ffa906d073e1df27b4bed987dd7252e5bc9-comps.xml -rw-r--r--. 1 apache apache 34443697 Oct 13 10:01 7a42eba03c74776be72deba4c94e2f0e9bdcacd5-filelists.xml.gz -rw-r--r--. 1 apache apache 14231598 Oct 13 10:01 0340275b6f931489c8663f6ea51b71d7e3eefc75-other.xml.gz -rw-r--r--. 1 apache apache 29411929 Oct 13 10:01 696ec2025734b26832dc980280c5390a494fbaa7-primary.xml.gz -rw-r--r--. 1 apache apache 93 Oct 13 10:01 0d11569ff8d032e17c471b13fe426ffc013c0039-updateinfo.xml.gz [root@sat repodata]# grep update repomd.xml <data type="updateinfo"><location href="repodata/0d11569ff8d032e17c471b13fe426ffc013c0039-updateinfo.xml.gz" /><timestamp>1476367266</timestamp><size>93</size><checksum type="sha1">0d11569ff8d032e17c471b13fe426ffc013c0039</checksum><open-size>58</open-size><open-checksum type="sha1">9d65415ae7cc4f506ce1efc0a2d8a955d10c5ac1</open-checksum></data>
Michael, does any of Will's comment make sense with what you understand?
I am surprised that it's so easily reproducible. When pulp does the copy from a pre-existing publication to the new one, it should be preserving the mtime and other attributes. I'll look at the code and try to confirm if that's the case. I'm also curious why pulp would be doing a publish when there are no errata in the repo (producing the 93B updateinfo file), and then do another publish after they've been added. Justin, can you shed some light on what workflow katello is walking through in Will's case, especially regarding pulp actions?
Michael, The action that I wrote out in comment #18 and that will performed in comment #20 was just to call a distributor publish for the yum distributor in pulp. Nothing else going on there. For new repos we do a publish immediately after creation so that any clients that may try to use the repo prior to syncing won't get a 404. Then the user syncs the rep, should get errata, and a 2nd publish should occur with the errata. This initial empty publish is only done at repo creation time, so In will comment, it doesn't make sense that it would be at play. I think that answers your question, let me know if not.
Ok, thanks. That does leave a mystery as to why Will saw two publishes happen for the same repo.
Just on the off chance, I ran through the same basic steps again today. I deleted the contents of /var/lib/pulp/published/yum/master/yum_distributor/Enterprise_Unix_Engineering-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server/* and then issued a sync through the WebUI instead of Foreman-rake console, This time around it only generated the one time stamp location, and it didn't generate additional updateinfo.xml.gz files. Just the one that is properly referenced. # ls -ltr published/yum/master/yum_distributor/Enterprise_Unix_Engineering-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server/1476451043.12/repodata/ total 80944 -rw-r--r--. 1 apache apache 2027 Oct 14 09:23 repomd.xml -rw-r--r--. 1 apache apache 34443697 Oct 14 09:23 9924d3f5014f5f1161c968ccc5c4f7c87caf1cc2-filelists.xml.gz -rw-r--r--. 1 apache apache 14231598 Oct 14 09:23 2a43ec80e0169f2ef9893d264faa3a3c6f757e3f-other.xml.gz -rw-r--r--. 1 apache apache 29411929 Oct 14 09:23 7f132c6f6312fac53eae1bb7a06a3c96d94685e9-primary.xml.gz -rw-r--r--. 1 apache apache 3676462 Oct 14 09:23 a5ffb9f6afdf9d03c4d56b5b694b93b916aff9eb-updateinfo.xml.gz lrwxrwxrwx. 1 apache apache 126 Oct 14 09:23 productid -> /var/lib/pulp/content/units/yum_repo_metadata_file/04/acbfa3dff8929de3fff11443183879e710ea7c1896d65bc728c270fae6f874/productid -rw-r--r--. 1 apache apache 1101933 Oct 14 09:23 54e59ffa906d073e1df27b4bed987dd7252e5bc9-comps.xml ATM I'm pub/promoting to my lab to validate that the clients will get the proper updateinfo.xml info as well.. If that works, I'm going to go back and repeat the process via foreman-rake console. I assume the flow is essentially the same via WebUI and the ForemanTasks.sync_task(Actions::Katello::Repository::MetadataGenerate, repo) command ?
An update from the client side. I completed publishing and promoting content to a remote capsule and a client beyond that. After publishing and promoting the content view to the approrpiate environment I issued a hammer capsule content sync to force the capsule to sync. A new repodata was generated, however it had the incorrect updateinfo.xml.gz w/ the 0 byte data referenced, and consequently, no updateinfo for clients # ls -ltr master/yum_distributor/Enterprise_Unix_Engineering-LAB-eue-rhel6-server-x86_64-cv-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server/1476454986.98/repodata/ total 80680 -rw-r--r--. 1 apache apache 2016 Oct 14 10:24 repomd.xml -rw-r--r--. 1 apache apache 93 Oct 14 10:24 6c0237c148097bcaf83a8af54cf557860f1cdc7a-updateinfo.xml.gz lrwxrwxrwx. 1 apache apache 126 Oct 14 10:24 productid -> /var/lib/pulp/content/units/yum_repo_metadata_file/7c/387007e99b1d6eaa7619ea5a48a4b327b668d9336dfdd6bf6fa3c743a396cf/productid -rw-r--r--. 1 apache apache 93 Oct 14 10:24 948d556d8681743727276c90bf60ab3a00d8acfb-updateinfo.xml.gz -rw-r--r--. 1 apache apache 1101933 Oct 14 10:24 081b2dd2e7bd78a7eb1b58018598b0391a6eca9c-comps.xml -rw-r--r--. 1 apache apache 3435058 Oct 14 10:24 58cd5adcb3782300972134659620367eba64da07-updateinfo.xml.gz -rw-r--r--. 1 apache apache 34438379 Oct 14 10:24 5b1c4ba453281f474cc5efccf3567d2fbd35f03b-filelists.xml.gz -rw-r--r--. 1 apache apache 14191430 Oct 14 10:24 a29ed52dc6fe16af1672ca3daf21af8b362da29b-other.xml.gz -rw-r--r--. 1 apache apache 29421380 Oct 14 10:24 cf5d3432d6bd57b81d4c892c1b1e2556019b31ab-primary.xml.gz -rw-r--r--. 1 apache apache 93 Oct 14 10:24 42b02590a2e367931cb6d1533c4c7693bf71fcf1-updateinfo.xml.gz <client>~]# yum clean all Loaded plugins: package_upload, product-id, search-disabled-repos, security, subscription-manager Cleaning repos: Enterprise_Unix_Engineering_EUE_Custom_Content_RHEL6_x86_64_eue_custom_content_rhel6_x86_64_repo rhel-6-server-rpms : rhel-6-server-satellite-tools-6.2-rpms Cleaning up Everything <client>~]# yum updateinfo list Loaded plugins: package_upload, product-id, search-disabled-repos, security, subscription-manager Enterprise_Unix_Engineering_EUE_Custom_Content_RHEL6_x86_64_eue_custom_content_rhel6_x86_64_repo | 2.1 kB 00:00 Enterprise_Unix_Engineering_EUE_Custom_Content_RHEL6_x86_64_eue_custom_content_rhel6_x86_64_repo/primary | 5.9 kB 00:00 Enterprise_Unix_Engineering_EUE_Custom_Content_RHEL6_x86_64_eue_custom_content_rhel6_x86_64_repo/updateinfo | 93 B 00:00 rhel-6-server-rpms | 2.0 kB 00:00 rhel-6-server-rpms/primary | 28 MB 00:00 rhel-6-server-rpms/updateinfo | 93 B 00:00 rhel-6-server-satellite-tools-6.2-rpms | 2.1 kB 00:00 rhel-6-server-satellite-tools-6.2-rpms/primary | 16 kB 00:00 rhel-6-server-satellite-tools-6.2-rpms/updateinfo | 93 B 00:00 Enterprise_Unix_Engineering_EUE_Custom_Content_RHEL6_x86_64_eue_custom_content_rhel6_x86_64_repo 11/11 rhel-6-server-rpms 18276/18276 rhel-6-server-satellite-tools-6.2-rpms 61/61 updateinfo list done After this I purged the existing timestamp location on the capsule, and then issued another hammer capsule sync. The second time the correct updateinfo.xml.gz file was created and referenced in the repomd.xml and the client was able to retrieve updateinfo from the capsule correctly. # ls -ltr master/yum_distributor/Enterprise_Unix_Engineering-LAB-eue-rhel6-server-x86_64-cv-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server/1476455750.35/repodata/ total 80644 -rw-r--r--. 1 apache apache 2027 Oct 14 10:40 repomd.xml -rw-r--r--. 1 apache apache 34436493 Oct 14 10:40 c1b9479294920cebdb72a3b133bda3a2b6646681-filelists.xml.gz -rw-r--r--. 1 apache apache 14170401 Oct 14 10:40 84dc31aa6ff8b1888e033d9e06b8fcd93d22557b-other.xml.gz -rw-r--r--. 1 apache apache 29410155 Oct 14 10:40 6a4e840ccc1206a6d35e5c5f6832f952bc4a1358-primary.xml.gz -rw-r--r--. 1 apache apache 3436923 Oct 14 10:40 af6d7595100e6bcf27d260aa13373a1f930083b8-updateinfo.xml.gz -rw-r--r--. 1 apache apache 1101933 Oct 14 10:40 081b2dd2e7bd78a7eb1b58018598b0391a6eca9c-comps.xml lrwxrwxrwx. 1 apache apache 126 Oct 14 10:40 productid -> /var/lib/pulp/content/units/yum_repo_metadata_file/7c/387007e99b1d6eaa7619ea5a48a4b327b668d9336dfdd6bf6fa3c743a396cf/productid <client>~]# yum clean all Loaded plugins: package_upload, product-id, search-disabled-repos, security, subscription-manager Cleaning repos: Enterprise_Unix_Engineering_EUE_Custom_Content_RHEL6_x86_64_eue_custom_content_rhel6_x86_64_repo rhel-6-server-rpms : rhel-6-server-satellite-tools-6.2-rpms Cleaning up Everything <client>~]# yum updateinfo list Loaded plugins: package_upload, product-id, search-disabled-repos, security, subscription-manager Enterprise_Unix_Engineering_EUE_Custom_Content_RHEL6_x86_64_eue_custom_content_rhel6_x86_64_repo | 2.1 kB 00:00 Enterprise_Unix_Engineering_EUE_Custom_Content_RHEL6_x86_64_eue_custom_content_rhel6_x86_64_repo/primary | 5.9 kB 00:00 Enterprise_Unix_Engineering_EUE_Custom_Content_RHEL6_x86_64_eue_custom_content_rhel6_x86_64_repo/updateinfo | 93 B 00:00 rhel-6-server-rpms | 2.0 kB 00:00 rhel-6-server-rpms/primary | 28 MB 00:00 rhel-6-server-rpms/updateinfo | 3.3 MB 00:00 rhel-6-server-satellite-tools-6.2-rpms | 2.1 kB 00:00 rhel-6-server-satellite-tools-6.2-rpms/primary | 16 kB 00:00 rhel-6-server-satellite-tools-6.2-rpms/updateinfo | 93 B 00:00 Enterprise_Unix_Engineering_EUE_Custom_Content_RHEL6_x86_64_eue_custom_content_rhel6_x86_64_repo 11/11 rhel-6-server-rpms 18276/18276 rhel-6-server-satellite-tools-6.2-rpms 61/61 RHBA-2016:1667 bugfix cronie-1.4.4-16.el6_8.2.x86_64 RHBA-2016:1667 bugfix cronie-anacron-1.4.4-16.el6_8.2.x86_64 RHBA-2016:0995 bugfix dracut-004-409.el6_8.2.noarch RHBA-2016:0995 bugfix dracut-kernel-004-409.el6_8.2.noarch RHBA-2016:2000 bugfix grep-2.20-5.el6_8.x86_64 RHBA-2016:1396 bugfix initscripts-9.03.53-1.el6_8.1.x86_64 RHBA-2016:1185 bugfix kernel-2.6.32-642.1.1.el6.x86_64 RHSA-2016:1406 Important/Sec. kernel-2.6.32-642.3.1.el6.x86_64 RHBA-2016:1185 bugfix kernel-firmware-2.6.32-642.1.1.el6.noarch RHSA-2016:1406 Important/Sec. kernel-firmware-2.6.32-642.3.1.el6.noarch RHBA-2016:1410 bugfix libcgroup-0.40.rc1-18.el6_8.x86_64 RHBA-2016:1659 bugfix libgudev1-147-2.73.el6_8.1.x86_64 RHBA-2016:1819 bugfix libgudev1-147-2.73.el6_8.2.x86_64 RHSA-2016:1547 Important/Sec. libtiff-3.9.4-18.el6_8.x86_64 RHBA-2016:1659 bugfix libudev-147-2.73.el6_8.1.x86_64 RHBA-2016:1819 bugfix libudev-147-2.73.el6_8.2.x86_64 RHBA-2016:1400 bugfix mdadm-3.3.4-1.el6_8.5.x86_64 RHBA-2016:1411 bugfix nfs-utils-1:1.2.3-70.el6_8.1.x86_64 RHBA-2016:2001 bugfix nfs-utils-1:1.2.3-70.el6_8.2.x86_64 RHBA-2016:1397 bugfix nss-softokn-3.14.3-23.3.el6_8.x86_64 RHBA-2016:1397 bugfix nss-softokn-freebl-3.14.3-23.3.el6_8.x86_64 RHBA-2016:0991 bugfix openssh-5.3p1-118.1.el6_8.x86_64 RHBA-2016:0991 bugfix openssh-clients-5.3p1-118.1.el6_8.x86_64 RHBA-2016:0991 bugfix openssh-server-5.3p1-118.1.el6_8.x86_64 RHSA-2016:0996 Important/Sec. openssl-1.0.1e-48.el6_8.1.x86_64 RHBA-2014:0778 bugfix ql2400-firmware-7.03.00-1.el6_5.noarch RHBA-2014:0777 bugfix ql2500-firmware-7.03.00-1.el6_5.noarch RHBA-2016:1408 bugfix tar-2:1.23-15.el6_8.x86_64 RHBA-2016:1666 bugfix tcsh-6.17-35.el6_8.1.x86_64 RHBA-2016:1659 bugfix udev-147-2.73.el6_8.1.x86_64 RHBA-2016:1819 bugfix udev-147-2.73.el6_8.2.x86_64 RHBA-2016:1398 bugfix yum-3.2.29-75.el6_8.noarch It seems that so long as there is a repodata with invalid updateinfo.xml.gz files in place, those will be propogated forward with new content view updates. The only recourse seems to be to manually remove the <timestamp> location and re-publish/promote/sync a capsule if you are experiencing issues. This might provie time consuming if there are a lot of content views :/
Still messing with this. Now its turned into what appears to be a pulp issue I think? I cleared the local repository repodata and re-sync'd w/ Red Hat. In the contents of the updateinfo.xml.gz file I can see newer Errata (eg RHBA-2016:2000 ) So Im getting Errata from Red Hat. However when I check the Errata that Satellite thinks it has, I can't find it (via hammer or the WebUI) [root@sat Enterprise_Unix_Engineering-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server]# cd 1476799525.45/repodata/ [root@sat repodata]# ls -la total 82140 drwxr-xr-x. 2 apache apache 4096 Oct 18 10:12 . drwxr-xr-x. 3 apache apache 966656 Oct 18 10:12 .. -rw-r--r--. 1 apache apache 29411929 Oct 18 10:12 14a5bd7294822b0310f07c6584ba9be421aff3eb-primary.xml.gz -rw-r--r--. 1 apache apache 1101933 Oct 18 10:12 54e59ffa906d073e1df27b4bed987dd7252e5bc9-comps.xml -rw-r--r--. 1 apache apache 3676462 Oct 18 10:12 5933fd03e3187a191ac1c3b2f06690e1f195a337-updateinfo.xml.gz -rw-r--r--. 1 apache apache 14231598 Oct 18 10:12 6da49856bb0ac82288df39d1361060a09ff57890-other.xml.gz -rw-r--r--. 1 apache apache 34443697 Oct 18 10:12 8d35198d695c40e57b7ae724fa10ca28c0429bc7-filelists.xml.gz lrwxrwxrwx. 1 apache apache 126 Oct 18 10:12 productid -> /var/lib/pulp/content/units/yum_repo_metadata_file/04/acbfa3dff8929de3fff11443183879e710ea7c1896d65bc728c270fae6f874/productid -rw-r--r--. 1 apache apache 2027 Oct 18 10:12 repomd.xml [root@sat repodata]# zcat 5933fd03e3187a191ac1c3b2f06690e1f195a337-updateinfo.xml.gz | grep 2016\:2000 <update from="release-engineering" status="final" type="bugfix" version="3"><id>RHBA-2016:2000</id><issued date="2016-10-04 19:55:42 UTC" /><title>grep bug fix update</title><release>0</release><rights>Copyright 2016 Red Hat Inc</rights><solution>Before applying this update, make sure all previously released errata relevant bug.</description><updated date="2016-10-04 19:55:54 UTC" /><references><reference href="https://access.redhat.com/errata/RHBA-2016:2000" id="RHBA-2016:2000" title="RHBA-2016:2000" type="self" /><reference href="https://bugzilla.redhat.com/show_bug.cgi?id=1367309" id="1367309" title="grep -r has been broken in RHEL 6.7" type="bugzilla" /></references><pkglist><collection short=""><name>rhel-6-server-rpms__6Server__x86_64</name><package arch="x86_64" epoch="0" name="grep" release="5.el6_8" src="grep-2.20-5.el6_8.src.rpm" version="2.20"><filename>grep-2.20-5.el6_8.x86_64.rpm</filename><sum type="sha256">bd62afcb201432da450b8029b680140f04310ec83d8913614d4ba6473e203597</sum></package></collection></pkglist></update> [root@sat repodata]# grep update repomd.xml <data type="updateinfo"><location href="repodata/5933fd03e3187a191ac1c3b2f06690e1f195a337-updateinfo.xml.gz" /><timestamp>1476799953</timestamp><size>3676462</size><checksum type="sha1">5933fd03e3187a191ac1c3b2f06690e1f195a337</checksum><open-size>21339236</open-size><open-checksum type="sha1">4f47b4e2fb849c88ded31e3fa6eb8a72eb86f024</open-checksum></data> [root@sat ~]# hammer erratum list --product-id 8 --organization-id 1 --repository-id 1 |grep 2016\:2000 [root@sat ~]#
The pulp team is working on this now. We are still looking for the root cause, but are making progress.
This issue could not be reproduced on upstream 2.8.7. I reproduced the issue on a satellite box that had pulp-rpm-plugins-2.8.7.3-1.el7sat.noarch installed on it. Through debugging I've confirmed this downstream-only commit[0] introduces this regression. Specifically it is L#76 here[1]. Upstream Pulp 2.8.7 does not have this code[0][1]. Instead you'll see this line in upstream 2.8.7 [2]. There was code similar to [0] that was added to upstream, and it was introduced in 2.9.0 as an optional feature as part of this upstream work [3]. I will talk with upstream Pulp devs about verifying that this feature is not present in 2.9+ to ensure that when satellite rebases onto a 2.9+ version of Pulp we will not regress. Also I wrote a reproducer for this which reproduces the issue 100%. I'll post that in the next comment. [0]: https://gitlab.sat.lab.tlv.redhat.com/satellite6/pulp-rpm/commit/6dca85b1a4df65ee59ec547c5fa6d33ab583c481 [1]: https://gitlab.sat.lab.tlv.redhat.com/satellite6/pulp-rpm/blob/2.8.7.3/plugins/pulp_rpm/plugins/distributors/yum/publish.py [2]: https://github.com/pulp/pulp_rpm/blob/2.8-release/plugins/pulp_rpm/plugins/distributors/yum/publish.py#L75 [3]: https://pulp.plan.io/issues/1543
Created attachment 1216790 [details] reproducer for this issue which uses pulp-admin Reproduces missing errata problem. Run the reproducer and it will create a repo directly in Pulp called 'zoo' and reproduce using that. The reproducer also checks and reports if the the Pulp installation is affected or not. You may need to add username and password specific to your sat6 installation of Pulp.
To fix this, the upstream commit needs to be cherry picked into satellite 2.8.7.x. Note that this code change won't fix affected repositories until they are published. Should Katello republish all repos as part of the errata also?
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
The Pulp upstream bug priority is at Urgent. Updating the external tracker on this bug.
Verified in Satellite 6.2.5 -bash-4.2# yum update --bugfix Loaded plugins: product-id, search-disabled-repos, subscription-manager Default_Organization_625_7_capsule | 2.1 kB 00:00:00 Default_Organization_625_7_tools | 2.1 kB 00:00:00 rhel-7-server-optional-rpms | 2.0 kB 00:00:00 rhel-7-server-rh-common-rpms | 2.1 kB 00:00:00 rhel-7-server-rpms | 2.0 kB 00:00:00 --> nss-tools-3.21.0-17.el7.x86_64 from @beaker-Server/7.3 removed (updateinfo) --> policycoreutils-2.5-8.el7.x86_64 from @beaker-Server/7.3 removed (updateinfo) --> nss-3.21.0-17.el7.x86_64 from @beaker-Server/7.3 removed (updateinfo) --> 32:bind-license-9.9.4-37.el7.noarch from @beaker-Server/7.3 removed (updateinfo) --> libgcrypt-1.5.3-13.el7_3.1.x86_64 from rhel-7-server-rpms removed (updateinfo) --> nss-sysinit-3.21.0-17.el7.x86_64 from @beaker-Server/7.3 removed (updateinfo) --> policycoreutils-2.5-9.el7.x86_64 from rhel-7-server-rpms removed (updateinfo) --> 32:bind-libs-lite-9.9.4-37.el7.x86_64 from @beaker-Server/7.3 removed (updateinfo) --> libgcrypt-1.5.3-12.el7_1.1.x86_64 from @beaker-Server/7.3 removed (updateinfo) --> nss-util-3.21.0-2.2.el7_2.x86_64 from @beaker-Server/7.3 removed (updateinfo) --> 32:bind-license-9.9.4-38.el7_3.noarch from rhel-7-server-rpms removed (updateinfo) --> nss-3.21.3-2.el7_3.x86_64 from rhel-7-server-rpms removed (updateinfo) --> nss-sysinit-3.21.3-2.el7_3.x86_64 from rhel-7-server-rpms removed (updateinfo) --> nss-tools-3.21.3-2.el7_3.x86_64 from rhel-7-server-rpms removed (updateinfo) --> nss-util-3.21.3-1.1.el7_3.x86_64 from rhel-7-server-rpms removed (updateinfo) --> 32:bind-libs-lite-9.9.4-38.el7_3.x86_64 from rhel-7-server-rpms removed (updateinfo) 13 package(s) needed (+0 related) for security, out of 21 available Resolving Dependencies --> Running transaction check ---> Package device-mapper.x86_64 7:1.02.135-1.el7 will be updated ---> Package device-mapper.x86_64 7:1.02.135-1.el7_3.1 will be an update ---> Package device-mapper-event.x86_64 7:1.02.135-1.el7 will be updated ---> Package device-mapper-event.x86_64 7:1.02.135-1.el7_3.1 will be an update ---> Package device-mapper-event-libs.x86_64 7:1.02.135-1.el7 will be updated ---> Package device-mapper-event-libs.x86_64 7:1.02.135-1.el7_3.1 will be an update ---> Package device-mapper-libs.x86_64 7:1.02.135-1.el7 will be updated ---> Package device-mapper-libs.x86_64 7:1.02.135-1.el7_3.1 will be an update ---> Package libgudev1.x86_64 0:219-30.el7 will be updated ---> Package libgudev1.x86_64 0:219-30.el7_3.6 will be an update ---> Package lvm2.x86_64 7:2.02.166-1.el7 will be updated ---> Package lvm2.x86_64 7:2.02.166-1.el7_3.1 will be an update ---> Package lvm2-libs.x86_64 7:2.02.166-1.el7 will be updated ---> Package lvm2-libs.x86_64 7:2.02.166-1.el7_3.1 will be an update ---> Package selinux-policy.noarch 0:3.13.1-102.el7 will be updated ---> Package selinux-policy.noarch 0:3.13.1-102.el7_3.4 will be an update ---> Package selinux-policy-targeted.noarch 0:3.13.1-102.el7 will be updated ---> Package selinux-policy-targeted.noarch 0:3.13.1-102.el7_3.4 will be an update ---> Package systemd.x86_64 0:219-30.el7 will be updated ---> Package systemd.x86_64 0:219-30.el7_3.6 will be an update ---> Package systemd-libs.x86_64 0:219-30.el7 will be updated ---> Package systemd-libs.x86_64 0:219-30.el7_3.6 will be an update ---> Package systemd-sysv.x86_64 0:219-30.el7 will be updated ---> Package systemd-sysv.x86_64 0:219-30.el7_3.6 will be an update ---> Package tzdata.noarch 0:2016g-2.el7 will be updated ---> Package tzdata.noarch 0:2016i-1.el7 will be an update --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================== Updating: device-mapper x86_64 7:1.02.135-1.el7_3.1 rhel-7-server-rpms 269 k device-mapper-event x86_64 7:1.02.135-1.el7_3.1 rhel-7-server-rpms 177 k device-mapper-event-libs x86_64 7:1.02.135-1.el7_3.1 rhel-7-server-rpms 177 k device-mapper-libs x86_64 7:1.02.135-1.el7_3.1 rhel-7-server-rpms 333 k libgudev1 x86_64 219-30.el7_3.6 rhel-7-server-rpms 76 k lvm2 x86_64 7:2.02.166-1.el7_3.1 rhel-7-server-rpms 1.1 M lvm2-libs x86_64 7:2.02.166-1.el7_3.1 rhel-7-server-rpms 984 k selinux-policy noarch 3.13.1-102.el7_3.4 rhel-7-server-rpms 412 k selinux-policy-targeted noarch 3.13.1-102.el7_3.4 rhel-7-server-rpms 6.4 M systemd x86_64 219-30.el7_3.6 rhel-7-server-rpms 5.2 M systemd-libs x86_64 219-30.el7_3.6 rhel-7-server-rpms 368 k systemd-sysv x86_64 219-30.el7_3.6 rhel-7-server-rpms 63 k tzdata noarch 2016i-1.el7 rhel-7-server-rpms 440 k Transaction Summary ============================================================================================================================================================================================== Upgrade 13 Packages Total download size: 16 M Is this ok [y/d/N]:
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:2940
*** Bug 1372360 has been marked as a duplicate of this bug. ***
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.