Bug 1125388
| Summary: | in some cases rpm publish of new repo fails once after upgrade from 2.3 | ||
|---|---|---|---|
| Product: | [Retired] Pulp | Reporter: | Chris Duryee <cduryee> |
| Component: | rpm-support | Assignee: | Chris Duryee <cduryee> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Preethi Thomas <pthomas> |
| Severity: | medium | Docs Contact: | |
| Priority: | high | ||
| Version: | 2.4 Beta | CC: | pthomas, rbarlow, skarmark |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | 2.5.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2014-11-24 21:33:58 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This happened to me today ☹ Here's an example of a "damaged" unit: In my database, I have this: …"_storage_path" : "/var/lib/pulp/content/rpm/Deployment_Guide-as-IN/5.8/1.el5/noarch/f3f4e807e531735d60429adca4ac6292ac802b80/Deployment_Guide-as-IN-5.8-1.el5.noarch.rpm"… But the file doesn't exist at that storage path. It exists in a subfolder of it called "Packages": $ ls -lah /var/lib/pulp/content/rpm/Deployment_Guide-as-IN/5.8/1.el5/noarch/f3f4e807e531735d60429adca4ac6292ac802b80/Packages/ total 8.3M drwxr-xr-x. 2 apache apache 4.0K Jun 2 20:13 . drwxr-xr-x. 3 apache apache 4.0K Jun 2 20:13 .. -rw-r--r--. 1 apache apache 8.3M Jun 2 20:13 Deployment_Guide-as-IN-5.8-1.el5.noarch.rpm I'm rerunning the sync/publish now. If you don't hear from me again, send hel^W^W^W^W^W^W^W^W you can assume that the resync/republish trick mentioned above worked! It's me again. Chris's suggestion to rerun sync/publish did seem to work. The publish succeeded, and the generated repository can be used. However, I noticed that the old packages are left around on the filesystem, and are not detected as orphans: [rbarlow@limeade ~]$ ls -lah /var/lib/pulp/content/rpm/Deployment_Guide-as-IN/5.8/1.el5/noarch/f3f4e807e531735d60429adca4ac6292ac802b80/ total 8.3M drwxr-xr-x. 3 apache apache 4.0K Aug 26 17:19 . drwxr-xr-x. 3 apache apache 4.0K Jun 2 20:13 .. -rw-r--r--. 1 apache apache 8.3M Aug 26 17:19 Deployment_Guide-as-IN-5.8-1.el5.noarch.rpm drwxr-xr-x. 2 apache apache 4.0K Jun 2 20:13 Packages [rbarlow@limeade ~]$ ls -lah /var/lib/pulp/content/rpm/Deployment_Guide-as-IN/5.8/1.el5/noarch/f3f4e807e531735d60429adca4ac6292ac802b80/Packages/ total 8.3M drwxr-xr-x. 2 apache apache 4.0K Jun 2 20:13 . drwxr-xr-x. 3 apache apache 4.0K Aug 26 17:19 .. -rw-r--r--. 1 apache apache 8.3M Jun 2 20:13 Deployment_Guide-as-IN-5.8-1.el5.noarch.rpm These two files are not hardlinks to the same file: [rbarlow@limeade ~]$ stat /var/lib/pulp/content/rpm/Deployment_Guide-as-IN/5.8/1.el5/noarch/f3f4e807e531735d60429adca4ac6292ac802b80/Deployment_Guide-as-IN-5.8-1.el5.noarch.rpm File: `/var/lib/pulp/content/rpm/Deployment_Guide-as-IN/5.8/1.el5/noarch/f3f4e807e531735d60429adca4ac6292ac802b80/Deployment_Guide-as-IN-5.8-1.el5.noarch.rpm' Size: 8677763 Blocks: 16952 IO Block: 4096 regular file Device: fd00h/64768d Inode: 1434364 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 48/ apache) Gid: ( 48/ apache) Access: 2014-08-26 17:19:49.388661930 -0400 Modify: 2014-08-26 17:19:51.683662375 -0400 Change: 2014-08-26 17:19:51.926662425 -0400 [rbarlow@limeade ~]$ stat /var/lib/pulp/content/rpm/Deployment_Guide-as-IN/5.8/1.el5/noarch/f3f4e807e531735d60429adca4ac6292ac802b80/Packages/Deployment_Guide-as-IN-5.8-1.el5.noarch.rpm File: `/var/lib/pulp/content/rpm/Deployment_Guide-as-IN/5.8/1.el5/noarch/f3f4e807e531735d60429adca4ac6292ac802b80/Packages/Deployment_Guide-as-IN-5.8-1.el5.noarch.rpm' Size: 8677763 Blocks: 16952 IO Block: 4096 regular file Device: fd00h/64768d Inode: 922068 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 48/ apache) Gid: ( 48/ apache) Access: 2014-08-26 12:07:55.171171763 -0400 Modify: 2014-06-02 20:13:51.241678922 -0400 Change: 2014-06-02 20:13:51.589678985 -0400 We will have to do something to clean this up for users who have upgraded. Unfortunately, it's not easy to detect whether files in /var/lib/content are part of units or not, because not every file in there has a storage path recorded in the database. We discussed this in-person yesterday. It appears that something may be changing _storage_path on the unit, which should not be happening. update: if you manually change the _storage_path of your units and move them on-disk to the new location then publish, the operation works as expected. Syncing repos that have units with non-standard paths also works as expected and does not create duplicate files. To repro this bz on a fresh 2.4 without upgrading, you need to manually change the _storage_path of your units, move the files to the new location, then add an additional repo that contains some units that were moved. This last step is the one that causes trouble. merged to master (should be in 2.5-dev as well once that is branched) build: 2.5.0-0.6.beta verified [root@cloud-qe-17 ~]# rpm -qa pulp-server pulp-server-2.5.0-0.11.beta.el6.noarch [root@cloud-qe-17 ~]# Upgraded from 2.3 ->2.5 with different types for content. pulp-manage-db ran without any errors had a rhel5 repo before the upgrade synced & published a second rhel5 repo without any errors |
Description of problem: If a 2.3 install with an RPM repo is upgraded to 2.4 and then a new repo is added that contains units that already exist, the first sync/publish of the new repo will fail. Subsequent sync/publishes will succeed. Version-Release number of selected component (if applicable): pulp-server-2.4.0-0.29.beta.el6.noarch pulp-rpm-plugins-2.4.0-0.29.beta.el6.noarch python-pulp-rpm-common-2.4.0-0.29.beta.el6.noarch How reproducible: every time Steps to Reproduce: 1. install 2.3 2. create an RPM repo A and sync from upstream source 3. upgrade to 2.4 4. optionally sync/publish repo A (success) 5. create repo B that contains some rpms in A 6. sync/publish B (fail) 7. sync/publish A (success) 8. sync/publish B again (success) Actual results: B's sync fails the first time with: "Will not create a symlink to a non-existent source" stack: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task R = retval = fun(*args, **kwargs) File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 306, in __call__ return super(Task, self).__call__(*args, **kwargs) File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__ return self.run(*args, **kwargs) File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/publish.py", line 98, in publish transfer_repo, conduit, call_config) File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/publish.py", line 126, in _do_publish publish_report = publish_repo(transfer_repo, conduit, call_config) File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 458, in wrap_f return f(*args, **kwargs) File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 143, in publish return self._publisher.publish() File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 323, in publish self.process_lifecycle() File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 92, in process_lifecycle step.process() File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 150, in process self._process_block() File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 562, in _process_block self.process_unit(package_unit) File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 386, in process_uni self._create_symlink(source_path, destination_path) File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 401, in _create_symlink raise RuntimeError(msg % {'s': source_path}) RuntimeError: Will not create a symlink to a non-existent source [/var/lib/pulp/content/rpm/akuma/1.7/3.el6/x86_64/131e2f61a72404481efe7ec80aaa6f64075e107bdbbbd4cf610dc219e6086902/akuma-1.7-3.el6.x86_64.rpm] Expected results: B's first sync/publish succeeds Additional info: I suspect this is related to the fix for #1093745. New units are created without a "Packages" dir and old units still have it. It looks like there may be an edge case related to assocating existing units created with 2.3 to new repos.