Bug 1125388 - in some cases rpm publish of new repo fails once after upgrade from 2.3
Summary: in some cases rpm publish of new repo fails once after upgrade from 2.3
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Pulp
Classification: Retired
Component: rpm-support
Version: 2.4 Beta
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: 2.5.0
Assignee: Chris Duryee
QA Contact: Preethi Thomas
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-07-31 18:00 UTC by Chris Duryee
Modified: 2014-11-24 21:33 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-11-24 21:33:58 UTC


Attachments (Terms of Use)

Description Chris Duryee 2014-07-31 18:00:17 UTC
Description of problem:

If a 2.3 install with an RPM repo is upgraded to 2.4 and then a new repo is added that contains units that already exist, the first sync/publish of the new repo will fail. Subsequent sync/publishes will succeed.

Version-Release number of selected component (if applicable): 
pulp-server-2.4.0-0.29.beta.el6.noarch
pulp-rpm-plugins-2.4.0-0.29.beta.el6.noarch
python-pulp-rpm-common-2.4.0-0.29.beta.el6.noarch


How reproducible: every time


Steps to Reproduce:
1. install 2.3
2. create an RPM repo A and sync from upstream source
3. upgrade to 2.4
4. optionally sync/publish repo A (success)
5. create repo B that contains some rpms in A
6. sync/publish B (fail)
7. sync/publish A (success)
8. sync/publish B again (success)

Actual results:

B's sync fails the first time with: "Will not create a symlink to a non-existent source"

stack:
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 306, in __call__
    return super(Task, self).__call__(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/publish.py", line 98, in publish
    transfer_repo, conduit, call_config)
  File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/publish.py", line 126, in _do_publish
    publish_report = publish_repo(transfer_repo, conduit, call_config)
  File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 458, in wrap_f
    return f(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 143, in publish
    return self._publisher.publish()
  File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 323, in publish
    self.process_lifecycle()
  File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 92, in process_lifecycle
    step.process()
  File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 150, in process
    self._process_block()
  File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 562, in _process_block
    self.process_unit(package_unit)
  File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 386, in process_uni
    self._create_symlink(source_path, destination_path)
  File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 401, in _create_symlink
    raise RuntimeError(msg % {'s': source_path})
RuntimeError: Will not create a symlink to a non-existent source [/var/lib/pulp/content/rpm/akuma/1.7/3.el6/x86_64/131e2f61a72404481efe7ec80aaa6f64075e107bdbbbd4cf610dc219e6086902/akuma-1.7-3.el6.x86_64.rpm]


Expected results: B's first sync/publish succeeds


Additional info: I suspect this is related to the fix for #1093745. New units are created without a "Packages" dir and old units still have it. It looks like there may be an edge case related to assocating existing units created with 2.3 to new repos.

Comment 2 Randy Barlow 2014-08-26 20:43:04 UTC
This happened to me today ☹

Here's an example of a "damaged" unit:

In my database, I have this:

…"_storage_path" : "/var/lib/pulp/content/rpm/Deployment_Guide-as-IN/5.8/1.el5/noarch/f3f4e807e531735d60429adca4ac6292ac802b80/Deployment_Guide-as-IN-5.8-1.el5.noarch.rpm"…

But the file doesn't exist at that storage path. It exists in a subfolder of it called "Packages":

$ ls -lah /var/lib/pulp/content/rpm/Deployment_Guide-as-IN/5.8/1.el5/noarch/f3f4e807e531735d60429adca4ac6292ac802b80/Packages/
total 8.3M
drwxr-xr-x. 2 apache apache 4.0K Jun  2 20:13 .
drwxr-xr-x. 3 apache apache 4.0K Jun  2 20:13 ..
-rw-r--r--. 1 apache apache 8.3M Jun  2 20:13 Deployment_Guide-as-IN-5.8-1.el5.noarch.rpm

I'm rerunning the sync/publish now. If you don't hear from me again, send hel^W^W^W^W^W^W^W^W you can assume that the resync/republish trick mentioned above worked!

Comment 3 Randy Barlow 2014-08-27 14:20:59 UTC
It's me again. Chris's suggestion to rerun sync/publish did seem to work. The publish succeeded, and the generated repository can be used. However, I noticed that the old packages are left around on the filesystem, and are not detected as orphans:

[rbarlow@limeade ~]$ ls -lah /var/lib/pulp/content/rpm/Deployment_Guide-as-IN/5.8/1.el5/noarch/f3f4e807e531735d60429adca4ac6292ac802b80/
total 8.3M
drwxr-xr-x. 3 apache apache 4.0K Aug 26 17:19 .
drwxr-xr-x. 3 apache apache 4.0K Jun  2 20:13 ..
-rw-r--r--. 1 apache apache 8.3M Aug 26 17:19 Deployment_Guide-as-IN-5.8-1.el5.noarch.rpm
drwxr-xr-x. 2 apache apache 4.0K Jun  2 20:13 Packages
[rbarlow@limeade ~]$ ls -lah /var/lib/pulp/content/rpm/Deployment_Guide-as-IN/5.8/1.el5/noarch/f3f4e807e531735d60429adca4ac6292ac802b80/Packages/
total 8.3M
drwxr-xr-x. 2 apache apache 4.0K Jun  2 20:13 .
drwxr-xr-x. 3 apache apache 4.0K Aug 26 17:19 ..
-rw-r--r--. 1 apache apache 8.3M Jun  2 20:13 Deployment_Guide-as-IN-5.8-1.el5.noarch.rpm

These two files are not hardlinks to the same file:

[rbarlow@limeade ~]$ stat /var/lib/pulp/content/rpm/Deployment_Guide-as-IN/5.8/1.el5/noarch/f3f4e807e531735d60429adca4ac6292ac802b80/Deployment_Guide-as-IN-5.8-1.el5.noarch.rpm 
  File: `/var/lib/pulp/content/rpm/Deployment_Guide-as-IN/5.8/1.el5/noarch/f3f4e807e531735d60429adca4ac6292ac802b80/Deployment_Guide-as-IN-5.8-1.el5.noarch.rpm'
  Size: 8677763   	Blocks: 16952      IO Block: 4096   regular file
Device: fd00h/64768d	Inode: 1434364     Links: 1
Access: (0644/-rw-r--r--)  Uid: (   48/  apache)   Gid: (   48/  apache)
Access: 2014-08-26 17:19:49.388661930 -0400
Modify: 2014-08-26 17:19:51.683662375 -0400
Change: 2014-08-26 17:19:51.926662425 -0400
[rbarlow@limeade ~]$ stat /var/lib/pulp/content/rpm/Deployment_Guide-as-IN/5.8/1.el5/noarch/f3f4e807e531735d60429adca4ac6292ac802b80/Packages/Deployment_Guide-as-IN-5.8-1.el5.noarch.rpm 
  File: `/var/lib/pulp/content/rpm/Deployment_Guide-as-IN/5.8/1.el5/noarch/f3f4e807e531735d60429adca4ac6292ac802b80/Packages/Deployment_Guide-as-IN-5.8-1.el5.noarch.rpm'
  Size: 8677763   	Blocks: 16952      IO Block: 4096   regular file
Device: fd00h/64768d	Inode: 922068      Links: 1
Access: (0644/-rw-r--r--)  Uid: (   48/  apache)   Gid: (   48/  apache)
Access: 2014-08-26 12:07:55.171171763 -0400
Modify: 2014-06-02 20:13:51.241678922 -0400
Change: 2014-06-02 20:13:51.589678985 -0400

We will have to do something to clean this up for users who have upgraded. Unfortunately, it's not easy to detect whether files in /var/lib/content are part of units or not, because not every file in there has a storage path recorded in the database.

Comment 4 Chris Duryee 2014-08-27 14:24:44 UTC
We discussed this in-person yesterday. It appears that something may be changing _storage_path on the unit, which should not be happening.

Comment 5 Chris Duryee 2014-08-28 17:37:49 UTC
update: if you manually change the _storage_path of your units and move them on-disk to the new location then publish, the operation works as expected. Syncing repos that have units with non-standard paths also works as expected and does not create duplicate files.

To repro this bz on a fresh 2.4 without upgrading, you need to manually change the _storage_path of your units, move the files to the new location, then add an additional repo that contains some units that were moved. This last step is the one that causes trouble.

Comment 6 Chris Duryee 2014-09-03 19:56:00 UTC
https://github.com/pulp/pulp_rpm/pull/556

Comment 7 Chris Duryee 2014-09-04 13:51:29 UTC
merged to master (should be in 2.5-dev as well once that is branched)

Comment 8 Chris Duryee 2014-09-30 13:52:49 UTC
build: 2.5.0-0.6.beta

Comment 9 Preethi Thomas 2014-10-23 17:22:24 UTC
verified
[root@cloud-qe-17 ~]# rpm -qa pulp-server
pulp-server-2.5.0-0.11.beta.el6.noarch
[root@cloud-qe-17 ~]# 


Upgraded from 2.3 ->2.5 with different types for content.

pulp-manage-db ran without any errors
had a rhel5 repo before the upgrade

synced & published a second rhel5 repo without any errors


Note You need to log in before you can comment on or make changes to this bug.