Bug 993644 - Applying migration pulp_rpm.migrations.0007_inventoried_custom_metadata failed.
Applying migration pulp_rpm.migrations.0007_inventoried_custom_metadata failed.
Status: CLOSED CURRENTRELEASE
Product: Pulp
Classification: Community
Component: z_other (Show other bugs)
2.1.1
x86_64 Linux
medium Severity medium
: ---
: ---
Assigned To: pulp-bugs
Preethi Thomas
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-06 06:53 EDT by Tim Hughes
Modified: 2016-03-29 12:20 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-11-07 18:03:33 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
rpm -qa|sort (18.47 KB, text/plain)
2013-08-06 06:53 EDT, Tim Hughes
no flags Details

  None (edit)
Description Tim Hughes 2013-08-06 06:53:46 EDT
Created attachment 783266 [details]
rpm -qa|sort

Description of problem:

yum upgrade from working pulp-server-2.1.1-1.el6.noarch to pulp-server-2.1.3-1.el6.noarch then ran pulp-manage-db. This took forever as it is doing an individual chown on every file. Then failed with the error in the the Additional info section. The actual file I believe that it is looking for is at:


[root@ld4repo02 ~]# ll /var/lib/pulp/content/isos/x86_64/CentOS-6.4-x86_64-netinstall.iso
-rw-r--r-- 1 apache apache 241172480 Jun  3 11:24 /var/lib/pulp/content/isos/x86_64/CentOS-6.4-x86_64-netinstall.iso



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.  
2.
3.

Actual results:
See error below

Expected results:
Clean upgrade and working pulp server at the end of the process

Additional info:

Serevr Details:
Linux ld4repo02.ld4.lmax 2.6.32-358.11.1.el6.x86_64 #1 SMP Wed Jun 12 03:34:52 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg_os-lv_root
                      15449552   3900852  10763908  27% /
tmpfs                  1961468         0   1961468   0% /dev/shm
/dev/vda1               516040     66880    422948  14% /boot
/dev/mapper/vg.pulp-mongodb
                      51606140  13008048  35976652  27% /var/lib/mongodb
/dev/mapper/vg.pulp-pulp
                     722486360 380351592 305434608  56% /var/lib/pulp


Software Installed:
See attached file

Error:
[root@repo02 ~]# pulp-manage-db
Beginning database migrations.
Migration package pulp.server.db.migrations is up to date at version 4
Migration package pulp_puppet.plugins.migrations is up to date at version 0
Applying pulp_rpm.migrations version 7
Applying migration pulp_rpm.migrations.0007_inventoried_custom_metadata failed.  See log for details.
2013-08-05 17:08:20,047 db:CRITICAL: Applying migration pulp_rpm.migrations.0007_inventoried_custom_metadata failed.
2013-08-05 17:08:20,059 db:CRITICAL: [Errno 2] No such file or directory: '/var/lib/pulp/working/repos/centos-6_4-os-x86_64/importers/isos/x86_64/CentOS-6.4-x86_64-netinstall.iso'
2013-08-05 17:08:20,126 db:CRITICAL: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/db/manage.py", line 79, in migrate_database
    update_current_version=not options.test)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/migrate/models.py", line 161, in apply_migration
    migration.migrate()
  File "/usr/lib/python2.6/site-packages/pulp_rpm/migrations/0007_inventoried_custom_metadata.py", line 43, in migrate
    inventory_custom_metadata()
  File "/usr/lib/python2.6/site-packages/pulp_rpm/migrations/0007_inventoried_custom_metadata.py", line 55, in inventory_custom_metadata
    migrate_repo(repo)
  File "/usr/lib/python2.6/site-packages/pulp_rpm/migrations/0007_inventoried_custom_metadata.py", line 100, in migrate_repo
    content_unit = create_content_unit(ftype_data, relative_path)
  File "/usr/lib/python2.6/site-packages/pulp_rpm/migrations/0007_inventoried_custom_metadata.py", line 128, in create_content_unit
    unit_data['_storage_path'] = get_content_storage_path(relative_path)
  File "/usr/lib/python2.6/site-packages/pulp_rpm/migrations/0007_inventoried_custom_metadata.py", line 143, in get_content_storage_path
    fix_owner(storage_dir)
  File "/usr/lib/python2.6/site-packages/pulp_rpm/migrations/0007_inventoried_custom_metadata.py", line 161, in fix_owner
    _recursive_fix_owner(start_path)
  File "/usr/lib/python2.6/site-packages/pulp_rpm/migrations/0007_inventoried_custom_metadata.py", line 159, in _recursive_fix_owner
    _recursive_fix_owner(os.path.join(path, c))
  File "/usr/lib/python2.6/site-packages/pulp_rpm/migrations/0007_inventoried_custom_metadata.py", line 159, in _recursive_fix_owner
    _recursive_fix_owner(os.path.join(path, c))
  File "/usr/lib/python2.6/site-packages/pulp_rpm/migrations/0007_inventoried_custom_metadata.py", line 159, in _recursive_fix_owner
    _recursive_fix_owner(os.path.join(path, c))
  File "/usr/lib/python2.6/site-packages/pulp_rpm/migrations/0007_inventoried_custom_metadata.py", line 159, in _recursive_fix_owner
    _recursive_fix_owner(os.path.join(path, c))
  File "/usr/lib/python2.6/site-packages/pulp_rpm/migrations/0007_inventoried_custom_metadata.py", line 159, in _recursive_fix_owner
    _recursive_fix_owner(os.path.join(path, c))
  File "/usr/lib/python2.6/site-packages/pulp_rpm/migrations/0007_inventoried_custom_metadata.py", line 159, in _recursive_fix_owner
    _recursive_fix_owner(os.path.join(path, c))
  File "/usr/lib/python2.6/site-packages/pulp_rpm/migrations/0007_inventoried_custom_metadata.py", line 159, in _recursive_fix_owner
    _recursive_fix_owner(os.path.join(path, c))
  File "/usr/lib/python2.6/site-packages/pulp_rpm/migrations/0007_inventoried_custom_metadata.py", line 154, in _recursive_fix_owner
    os.chown(path, apache_uid, apache_gid)
OSError: [Errno 2] No such file or directory: '/var/lib/pulp/working/repos/centos-6_4-os-x86_64/importers/isos/x86_64/CentOS-6.4-x86_64-netinstall.iso'

Database migrations complete.
Loading content types.
Content types loaded.
Comment 1 Tim Hughes 2013-08-06 11:47:41 EDT
the file it is trying to chown is a symlink. This is what it looks like

[root@ld4repo02 ~]# ll /var/lib/pulp/working/repos/centos-6_4-os-x86_64/importers/isos/x86_64/CentOS-6.4-x86_64-netinstall.iso
lrwxrwxrwx 1 apache apache 122 Jun 20 08:57 /var/lib/pulp/working/repos/centos-6_4-os-x86_64/importers/isos/x86_64/CentOS-6.4-x86_64-netinstall.iso -> ../../../../../../../../../../content/distribution/ks-CentOS-6.4-x86_64/../../isos/x86_64/CentOS-6.4-x86_64-netinstall.iso
Comment 2 Tim Hughes 2013-08-06 12:05:02 EDT
A little more information. 


[root@ld4repo02 x86_64]# pwd
/var/lib/pulp/working/repos/centos-6_4-os-x86_64/importers/isos/x86_64
[root@ld4repo02 x86_64]# ll
total 4
lrwxrwxrwx 1 apache apache 122 Jun 20 08:57 CentOS-6.4-x86_64-netinstall.iso -> ../../../../../../../../../../content/distribution/ks-CentOS-6.4-x86_64/../../isos/x86_64/CentOS-6.4-x86_64-netinstall.iso
[root@ld4repo02 x86_64]# ll ../../../../../../../../../../content/distribution/ks-CentOS-6.4-x86_64/../../isos/x86_64/CentOS-6.4-x86_64-netinstall.iso
ls: cannot access ../../../../../../../../../../content/distribution/ks-CentOS-6.4-x86_64/../../isos/x86_64/CentOS-6.4-x86_64-netinstall.iso: No such file or directory


If i remove several "../" from the start of where the symlink is pointing we can get to the correct file.

[root@ld4repo02 x86_64]# ll ../../../../../../content/distribution/ks-CentOS-6.4-x86_64/../../isos/x86_64/CentOS-6.4-x86_64-netinstall.iso
-rw-r--r-- 1 apache apache 241172480 Jun  3 11:24 ../../../../../../content/distribution/ks-CentOS-6.4-x86_64/../../isos/x86_64/CentOS-6.4-x86_64-netinstall.iso




Going to try fixing the symlinks by hand and see where we get to.


[root@ld4repo02 x86_64]# pwd
/var/lib/pulp/working/repos/centos-6_4-os-x86_64/importers/isos/x86_64
[root@ld4repo02 x86_64]# rm CentOS-6.4-x86_64-netinstall.iso
rm: remove symbolic link `CentOS-6.4-x86_64-netinstall.iso'? y
[root@ld4repo02 x86_64]# ln -s ../../../../../../content/distribution/ks-CentOS-6.4-x86_64/../../isos/x86_64/CentOS-6.4-x86_64-netinstall.iso CentOS-6.4-x86_64-netinstall.iso
[root@ld4repo02 x86_64]# file CentOS-6.4-x86_64-netinstall.iso
CentOS-6.4-x86_64-netinstall.iso: symbolic link to `../../../../../../content/distribution/ks-CentOS-6.4-x86_64/../../isos/x86_64/CentOS-6.4-x86_64-netinstall.iso'
[root@ld4repo02 x86_64]# file ../../../../../../content/distribution/ks-CentOS-6.4-x86_64/../../isos/x86_64/CentOS-6.4-x86_64-netinstall.iso
../../../../../../content/distribution/ks-CentOS-6.4-x86_64/../../isos/x86_64/CentOS-6.4-x86_64-netinstall.iso: ISO 9660 CD-ROM filesystem data 'CentOS                         ' (bootable)
[root@ld4repo02 x86_64]#
Comment 3 Tim Hughes 2013-08-07 09:35:09 EDT
OK , changing the above symlinks allowed the migration to progress to the end.
Comment 4 Michael Hrivnak 2013-09-03 14:47:59 EDT
There seems to be something fishy going on. That migration should only ever apply to repositories that have the yum importer associated, but an ISO repo should have only the iso_importer. Repos can only have one importer at a time.

Can you run the following command and give us the output for the repo that is experiencing this problem?

$ pulp-admin iso repo list --details

That will show us which importer is associated.
Comment 5 Michael Hrivnak 2013-09-05 14:11:58 EDT
I think this may be related to the centos repository metadata linking to content that is outside the base path of the repository. That strikes me as weird, and I'm not surprised it would cause problems. Have a look here:

http://mirror.rackspace.com/CentOS/6.4/os/x86_64/.treeinfo

Note the "boot.iso" line.
Comment 6 Sayli Karmarkar 2013-09-20 00:48:22 EDT
Moving back to NEW as we are waiting on more info from Tim.

Tim, can you provide more information for comment 4 and 5?
Comment 7 Chris Duryee 2014-11-07 18:03:33 EST
The CentOS 6.6 treeinfo metadata appears to not link to files outside of the repo anymore:

http://mirror.rackspace.com/CentOS/6.6/os/x86_64/.treeinfo

I am marking this bug as CLOSED/CURRENTRELEASE since we have not had other reports of the same issue. If you are still encountering issues, feel free to re-open this bug.
Comment 8 Michael Hrivnak 2016-03-29 12:20:05 EDT
I don't think we need additional info anymore.

Note You need to log in before you can comment on or make changes to this bug.