Hide Forgot
Description of problem: When i try to create a repo with symlinks attribute and local feed, i am unable to synchronize it. It found no package. It works without symlinks flag Version-Release number of selected component (if applicable): pulp-0.0.237-2.el6.noarch How reproducible: # pulp-admin repo create --id rhel5s-i386-hts --feed file:///var/mrepo/rhel5s-i386/hts Successfully created repository [ rhel5s-i386-hts ] # pulp-admin repo sync --id rhel5s-i386-hts -F Sync for repository rhel5s-i386-hts started Sync: Finished 5/5 new items downloaded 0/5 existing items processed Item Details: Rpm: 5/5 # pulp-admin repo delete --id rhel5s-i386-hts Successful deleted repository [ rhel5s-i386-hts ] # pulp-admin repo create --id rhel5s-i386-hts --feed file:///var/mrepo/rhel5s-i386/hts --symlinks Successfully created repository [ rhel5s-i386-hts ] # pulp-admin repo sync --id rhel5s-i386-hts -F Sync for repository rhel5s-i386-hts started You can safely CTRL+C this current command and it will continue Waiting / Step: Importing data into pulp Waiting - Sync: Finished 0/0 new items downloaded 0/0 existing items processed Additional info: I try to follow the code, and it seems symlinks to be unclear to me * 'repo create --symlinks' actually create a directory in /var/lib/pulp/repos. symlinks seems have no effect * 'repo sync' will try to create a symlink over existing directory. This failed, but as code does not test if target directory already exist, this does not work. (see synchronizers.py, method local near line 874) If i manually delete /var/lib/pulp/repos/rhel5s-i386-hts directory and re sync, it create the symlinks, but still failed with backtrace This is because the mrepo directory does not include the metadata here. Another test to include... 2011-10-09 17:15:54,067 26564:140715514021632: pulp.server.util:DEBUG: util:426 Create symlink for [../../../mrepo/rhel5s-i386/hts] to [/var/lib/pulp//repos/rhel5s-i386-hts] 2011-10-09 17:15:54,125 26564:140715514021632: pulp.server.api.synchronizers:INFO: synchronizers:202 Begin to add packages from /var/lib/pulp//repos/rhel5s-i386-hts into rhel5s-i386 -hts 2011-10-09 17:15:54,505 26564:140715514021632: pulp.server.tasking.task:ERROR: task:391 Task failed: Task 94aeb300-f289-11e0-a813-000c29d2bba6: _sync(rhel5s-i386-hts, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7ffadc1e9e10>, skip_dict={}, max_speed=None, threads=None, progress_callback=<bound method RepoSyncTask.progress_callback of <pulp.server.api.repo_sync_task.RepoSyncTask object at 0x7ffb1894e550>>) Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 340, in run result = self.callable(*self.args, **self.kwargs) File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 262, in _sync progress_callback, synchronizer, max_speed, threads) File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 345, in fetch_content added_packages = synchronizer.add_packages_from_dir(repo_dir, repo_id, skip_dict) File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 203, in add_packages_from_dir package_list = pulp.server.util.get_repo_packages(dir) File "/usr/lib/python2.6/site-packages/pulp/server/util.py", line 302, in get_repo_packages sack.populate(r, 'metadata', None, 0) File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 164, in populate if self._check_db_version(repo, mydbtype): File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 222, in _check_db_version return repo._check_db_version(mdtype) File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1255, in _check_db_version repoXML = self.repoXML File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1454, in <lambda> repoXML = property(fget=lambda self: self._getRepoXML(), File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1450, in _getRepoXML raise Errors.RepoError, msg RepoError: Cannot retrieve repository metadata (repomd.xml) for repository: /tmp/temp_pulp_repotwAQeP. Please verify its path and try again If i change the feed to point on mrepo published directory (with metadata), this doesn't crash, but still does not work, although logs (/var/log/pulp/pulp/log) says it found packages. # pulp-admin repo create --id rhel5s-i386-hts --feed file:///var/www/mrepo/rhel5s-i386/RPMS.hts/ --symlinks # rm -rf /var/lib/pulp/repos/rhel5s-i386-hts # pulp-admin repo sync --id rhel5s-i386-hts -F Sync: Finished 0/0 new items downloaded 0/0 existing items processed 2011-10-09 17:41:28,087 26564:140715514021632: pulp.server.api.repo_sync:INFO: repo_sync:244 Sync invoked for repo <rhel5s-i386-hts> 2011-10-09 17:41:28,090 26564:140715514021632: pulp.server.api.repo_sync:INFO: repo_sync:258 Sync of rhel5s-i386-hts starting, skip_dict = {} 2011-10-09 17:41:28,094 26564:140715514021632: pulp.server.api.synchronizers:INFO: synchronizers:854 sync of /var/www/mrepo/rhel5s-i386/RPMS.hts/ for repo rhel5s-i386-hts 2011-10-09 17:41:28,095 26564:140715514021632: pulp.server.api.synchronizers:INFO: synchronizers:645 Found 5 packages in /var/www/mrepo/rhel5s-i386/RPMS.hts/ 2011-10-09 17:41:28,096 26564:140715514021632: pulp.server.api.synchronizers:INFO: synchronizers:657 Found 0 delta rpm packages in /var/www/mrepo/rhel5s-i386/RPMS.hts/ 2011-10-09 17:41:28,096 26564:140715514021632: pulp.server.api.synchronizers:INFO: synchronizers:874 create a symlink to src directory /var/www/mrepo/rhel5s-i386/RPMS.hts/ /var/lib/pulp//repos/rhel5s-i386-hts 2011-10-09 17:41:28,097 26564:140715514021632: pulp.server.util:DEBUG: util:426 Create symlink for [../../../www/mrepo/rhel5s-i386/RPMS.hts] to [/var/lib/pulp//repos/rhel5s-i386-hts] 2011-10-09 17:41:28,158 26564:140715514021632: pulp.server.api.synchronizers:INFO: synchronizers:202 Begin to add packages from /var/lib/pulp//repos/rhel5s-i386-hts into rhel5s-i386-hts 2011-10-09 17:41:28,322 26564:140715514021632: pulp.server.api.synchronizers:DEBUG: synchronizers:204 Processing 5 potential packages 2011-10-09 17:41:48,648 26564:140715514021632: pulp.server.api.synchronizers:INFO: synchronizers:211 Repo: rhel5s-i386-hts read [5] packages took 20.4893498421 seconds 2011-10-09 17:41:48,653 26564:140715514021632: pulp.server.api.synchronizers:DEBUG: synchronizers:292 Processing any images synced as part of the repo 2011-10-09 17:41:48,654 26564:140715514021632: pulp.server.api.synchronizers:INFO: synchronizers:295 No image files to import to repo.. 2011-10-09 17:41:48,657 26564:140715514021632: pulp.server.api.synchronizers:DEBUG: synchronizers:272 Processing any files synced as part of the repo 2011-10-09 17:41:48,657 26564:140715514021632: pulp.server.api.synchronizers:INFO: synchronizers:275 No metadata for 'File Sync' present; no files to import to repo.. 2011-10-09 17:41:48,665 26564:140715514021632: pulp.server.api.synchronizers:DEBUG: synchronizers:239 repodata has filetypes of ['filelists', 'primary', 'primary_db', 'other_db', 'other', 'filelists_db'] 2011-10-09 17:41:48,665 26564:140715514021632: pulp.server.api.synchronizers:INFO: synchronizers:267 Skipping errata imports from sync process 2011-10-09 17:41:48,667 26564:140715514021632: pulp.server.api.repo_sync:INFO: repo_sync:265 Sync returned 5 packages, 0 errata in 20.5772628784 seconds 2011-10-09 17:41:48,669 26564:140715514021632: pulp.server.api.repo_sync:INFO: repo_sync:276 0 old packages to process, 5 new packages to process 2011-10-09 17:41:48,670 26564:140715514021632: pulp.server.api.repo:DEBUG: repo:971 remove_packages invoked on rhel5s-i386-hts with no packages 2011-10-09 17:41:48,752 26564:140715514021632: pulp.server.api.repo_sync:INFO: repo_sync:291 Examining 0 errata from repo rhel5s-i386-hts 2011-10-09 17:41:48,753 26564:140715514021632: pulp.server.api.repo_sync:INFO: repo_sync:295 Removing 0 old errata from repo rhel5s-i386-hts 2011-10-09 17:41:48,758 26564:140715514021632: pulp.server.api.repo:DEBUG: repo:971 remove_packages invoked on rhel5s-i386-hts with no packages 2011-10-09 17:41:48,760 26564:140715514021632: pulp.server.api.repo_sync:INFO: repo_sync:304 Adding 0 new errata to repo rhel5s-i386-hts 2011-10-09 17:41:48,765 26564:140715514021632: pulp.server.tasking.task:INFO: task:376 Task succeeded: Task 270ac178-f28d-11e0-906c-000c29d2bba6: _sync(rhel5s-i386-hts, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7ffadc9ba950>, skip_dict={}, max_speed=None, threads=None, progress_callback=<bound method RepoSyncTask.progress_callback of <pulp.server.api.repo_sync_task.RepoSyncTask object at 0x7ffadc9afa10>>) Is this correct ? metadata have to be recreated ? I think they are not recreated in this case, where ? inside the symlink (but you will overwrite actual metadata) ? or outside the symlink (where?) ?
Also, if i try to delete repo with manually created symlink, it does not remove symlink. And if i try to recreate repo, it failed (due to permission error), but repo is created (even if previous operation failed) # pulp-admin repo delete --id rhel5s-i386-hts Successful deleted repository [ rhel5s-i386-hts ] # ll total 32 drwxr-xr-x. 5 apache apache 4096 9 oct. 10:27 centos5-os-i386 drwxr-xr-x. 5 apache apache 4096 9 oct. 10:43 centos5-os-x86_64 drwxr-xr-x. 4 apache apache 4096 9 oct. 10:11 centos5-updates-i386 drwxr-xr-x. 4 apache apache 4096 9 oct. 10:29 centos5-updates-x86_64 drwxr-xr-x. 5 apache apache 4096 9 oct. 11:07 centos6-os-i386 drwxr-xr-x. 5 apache apache 4096 9 oct. 11:42 centos6-os-x86_64 drwxr-xr-x. 5 apache apache 4096 9 oct. 10:50 centos6-updates-i386 drwxr-xr-x. 5 apache apache 4096 9 oct. 11:17 centos6-updates-x86_64 lrwxrwxrwx. 1 apache apache 39 9 oct. 18:08 rhel5s-i386-hts -> ../../../www/mrepo/rhel5s-i386/RPMS.hts # pulp-admin repo create --id rhel5s-i386-hts --feed file:///var/www/mrepo/rhel5s-i386/RPMS.hts --symlinks error: operation failed: OSError: [Errno 13] Permission denied: '/var/lib/pulp//repos/rhel5s-i386-hts/repodata.old' # echo $? 244 # pulp-admin repo delete --id rhel5s-i386-hts Successful deleted repository [ rhel5s-i386-hts ]
Laurent, Thanks for a detailed bug report. Since the time symlink support for local feed repositories was added, there have been a lot of changes in pulp code to improve efficiency in that area. One of the major change that affects this feature is centralized packages and file storage. Packages and files in all repositories are now stored under /var/lib/pulp/packages and /var/lib/pulp/files. Because of this change, content for local repos can no more be directly linked with symlink. Local content will be copied under centralized content location and then symlinked from repo under /var/lib/pulp/repos. Considering there is not much benefit from symlink even if we fix it as described above, we are thinking about dropping this feature in next release, unless there is a solid use case for symlink feature for local repositories.
commit d01eedd3d9ace064c3041f3b790bb0191e9ba6cd Author: Sayli Karmarkar <skarmark> Date: Fri Oct 28 14:11:09 2011 -0700 744587 - removed use_symlinks flag from db and symlinks flag from pulp api and cli
build: 0.244
verified [root@preethi ~]# pulp-admin -u admin -p admin repo create --preserve_metadata --id=local --feed=file:///mnt/disk --symlinks --help Usage: pulp-admin <options> repo create <options> pulp-admin: error: no such option: --symlinks [root@preethi ~]# pulp-admin -u admin -p admin repo create --preserve_metadata --id=local --feed=file:///mnt/disk --help Usage: pulp-admin <options> repo create <options> Options: -h, --help show this help message and exit --id=ID repository id (required) --name=NAME common repository name --arch=ARCH package arch the repository should support --feed=FEED url feed to populate the repository --feed_ca=FEED_CA path location to the feed's ca certificate --feed_cert=FEED_CERT path location to the feed's entitlement certificate --feed_key=FEED_KEY path location to the feed's entitlement certificate key --consumer_ca=CONSUMER_CA path location to the ca certificate used to verify consumer requests --consumer_cert=CONSUMER_CERT path location to the entitlement certificate consumers will be provided at bind to grant access to this repo --consumer_key=CONSUMER_KEY path location to the consumer entitlement certificate key --relativepath=RELATIVEPATH relative path where the repository is stored and exposed to clients; this defaults to feed path if not specified --groupid=GROUPID a group to which the repository belongs; this is just a string identifier --gpgkeys=KEYS a ',' separated list of directories and/or files containing GPG keys --checksum_type=CHECKSUM_TYPE checksum type to use when yum metadata is generated for this repo; default:sha256 --notes=NOTES additional information about repo in a dictionary form inside a string --preserve_metadata Preserves the original metadata; only works with feed repos --content_type=CONTENT_TYPE content type allowed in this repository; default:yum; supported: [yum, file] [root@preethi ~]#
Pulp v1.0 is released Closed Current Release.