We need to a call to get file checksum from the server. This helps us to skip existing files or detect file conflicts (checksum mismatch) before the upload even starts. Since pulp is supposed to support non-rpm content (which means huge files), it would be nice to return also file sizes - to check file sizes before checksums come into play. the call could look like: get_file_checksums({"repo_id1": ["file_name", ...], "repo_id2": [], ...}) -> { "repo_id1": { "file_name": [file_size, "checksum"], "file_name": [file_size, "checksum"], ... }, "repo_id2": ... } If a file doesn't exist on server, it's not returned in the result. I'm not sure whether the same checksum type is used in all repos, otherwise a checksum type needs to be returned as well.
*** Bug 636525 has been marked as a duplicate of this bug. ***
This is done as part of sprint16 effort. Moving to on_QA.
What's the implemented call name and arguments? Is the code included in pulp repo already?
Prad, Could you reply for the above comment?
I think this bug was accidentally moved to ON_QA during the mass move for sprint16. The call is in place on server but the webservices handler doesnt expose it yet. So clients dont have the ability to look up checksum info per rpm. Moving back to assigned. should be a quick change to expose this.
Now alternately you can perform the following, which has the chekcsum info: $ sudo pulp-admin package info --name=zsh --repoid=rhel-server-5Server-i386 +------------------------------------------+ Package Information +------------------------------------------+ vendor: None name: zsh repo_defined: True checksum: {u'sha256': u'37a41efec568ddd4f4afd153ee824050355f87f1'} requires: [u'libc.so.6(GLIBC_2.4)', u'libm.so.6(GLIBC_2.0)', u'rtld(GNU_HASH)', u'libdl.so.2(GLIBC_2.1)', u'libc.so.6(GLIBC_2.1.3)', u'libnsl.so.1', u'fileutils', u'libm.so.6', u'libc.so.6(GLIBC_2.2)', u'libc.so.6', u'libdl.so.2', u'libc.so.6(GLIBC_2.3)', u'libc.so.6(GLIBC_2.3.4)', u'/bin/sh', u'libc.so.6(GLIBC_2.1)', u'libtermcap.so.2', u'/sbin/install-info', u'libc.so.6(GLIBC_2.0)', u'libdl.so.2(GLIBC_2.0)', u'libcap', u'config(zsh)', u'grep'] download_url: https://localhost//pulp/repos/rhel-server-5Server-i386/zsh-4.2.6-1.i386.rpm filename: zsh-4.2.6-1.i386.rpm epoch: 0 version: 4.2.6 provides: [u'zle.so', u'zprof.so', u'zleparameter.so', u'rlimits.so', u'stat.so', u'zutil.so', u'clone.so', u'deltochar.so', u'mapfile.so', u'sched.so', u'termcap.so', u'cap.so', u'datetime.so', u'zselect.so', u'zpty.so', u'complete.so', u'tcp.so', u'system.so', u'compctl.so', u'files.so', u'socket.so', u'zftp.so', u'example.so', u'config(zsh)', u'complist.so', u'parameter.so', u'computil.so', u'zsh', u'mathfunc.so', u'langinfo.so'] _ns: packages release: 1 _id: 3ac9143e-2f9f-4a1d-88aa-8e7d855f62aa arch: i386 id: 3ac9143e-2f9f-4a1d-88aa-8e7d855f62aa description: The zsh shell is a command interpreter usable as an interactive login shell and as a shell script command processor. Zsh resembles the ksh shell (the Korn shell), but includes many enhancements. Zsh supports command line editing, built-in spelling correction, programmable command completion, shell functions (with autoloading), a history Would this be sufficient or do you prefer a separate call?
fixed! The New API call is on RepoApi called get_file_checksum() with same footprint as you requested. To test: $cat test.py #! /usr/bin/python from pulp.server.db import connection connection.initialize() from pulp.server import auditing auditing.initialize() from pulp.server.api.package import PackageApi from pulp.server.api.repo import RepoApi p = PackageApi() print "Lookup package checksum PackageApi.package_checksum:\n" print p.package_checksum('skytux-repo-1.0-4.noarch.rpm') r = RepoApi() print "Lookup batch repo-package RepoApi.get_file_checksum:\n" print r.get_file_checksums({'foo123' : ['skytux-repo-1.0-4.noarch.rpm'], 'foo1234' : ['pyxf86config-0.3.37-10.fc14.x86_64.rpm', 'foo']}) $ sudo ./test.py Lookup package checksum PackageApi.package_checksum: [SON([(u'checksum', SON([(u'sha256', u'fa6c87a7d7cdab86e1ef43b1f596ecff35854f1863353f130b197fc549336def')])), (u'_id', u'd5cbc4c0-ca0c-4f8b-8cc3-64b4d717cc09')])] Lookup batch repo-package RepoApi.get_file_checksum: {'foo1234': {'pyxf86config-0.3.37-10.fc14.x86_64.rpm': [SON([(u'sha256', u'9b72be60ab2310916e1fbadcc8acdfca32b3d742e7e9ec55a3555bea89554214')])], 'foo': []}, 'foo123': {'skytux-repo-1.0-4.noarch.rpm': [SON([(u'sha256', u'fa6c87a7d7cdab86e1ef43b1f596ecff35854f1863353f130b197fc549336def')])]}}
Note this does not include filesize yet. We dont have that info in our db yet, once we have that, I'll include that info in the same call.
Fixed in build 0.122.
[root@preethi ~]# pulp-admin package info --name=zsh --repoid=f14 +------------------------------------------+ Package Information +------------------------------------------+ vendor: None name: zsh repo_defined: True checksum: {'sha256': 'ac64b95a3c879a27c5893bf0403b5734b41bd4abf73864383ec71f39ca12ce43'} download_url: https://preethi.usersys.redhat.com//pulp/repos/f14/zsh-4.3.10-5.fc14.x86_64.rpm filename: zsh-4.3.10-5.fc14.x86_64.rpm epoch: 0 version: 4.3.10 arch: x86_64 provides: ['zsh(x86-64)', 'zsh', 'zutil.so()(64bit)', 'zselect.so()(64bit)', 'zpty.so()(64bit)', 'zprof.so()(64bit)', 'zleparameter.so()(64bit)', 'zle.so()(64bit)', 'zftp.so()(64bit)', 'terminfo.so()(64bit)', 'termcap.so()(64bit)', 'tcp.so()(64bit)', 'system.so()(64bit)', 'stat.so()(64bit)', 'socket.so()(64bit)', 'sched.so()(64bit)', 'rlimits.so()(64bit)', 'regex.so()(64bit)', 'parameter.so()(64bit)', 'newuser.so()(64bit)', 'mathfunc.so()(64bit)', 'mapfile.so()(64bit)', 'langinfo.so()(64bit)', 'files.so()(64bit)', 'example.so()(64bit)', 'deltochar.so()(64bit)', 'datetime.so()(64bit)', 'curses.so()(64bit)', 'config(zsh)', 'computil.so()(64bit)', 'complist.so()(64bit)', 'complete.so()(64bit)', 'compctl.so()(64bit)', 'clone.so()(64bit)', 'cap.so()(64bit)', 'attr.so()(64bit)'] _ns: packages release: 5.fc14 _id: 201e12f3-143b-4697-b1cc-d3b98a32610d requires: ['/bin/sh', 'libdl.so.2()(64bit)', 'mktemp', 'libncursesw.so.5()(64bit)', 'fileutils', 'libm.so.6(GLIBC_2.2.5)(64bit)', 'libc.so.6(GLIBC_2.3.4)(64bit)', 'libc.so.6(GLIBC_2.4)(64bit)', 'libc.so.6(GLIBC_2.3)(64bit)', 'grep', 'libc.so.6(GLIBC_2.2.5)(64bit)', 'libc.so.6()(64bit)', 'libtinfo.so.5()(64bit)', 'rtld(GNU_HASH)', 'coreutils', 'libc.so.6(GLIBC_2.7)(64bit)', '/sbin/install-info', '/bin/sh', 'libdl.so.2(GLIBC_2.2.5)(64bit)', 'libm.so.6()(64bit)', 'grep', 'libc.so.6(GLIBC_2.11)(64bit)', '/sbin/install-info'] id: 201e12f3-143b-4697-b1cc-d3b98a32610d description: The zsh shell is a command interpreter usable as an interactive login shell and as a shell script command processor. Zsh resembles the ksh shell (the Korn shell), but includes many enhancements. Zsh supports command line editing, built-in spelling correction, programmable command completion, shell functions (with autoloading), a history mechanism, and more.
[root@preethi ~]# ./test.py Lookup package checksum PackageApi.package_checksum: [SON([(u'checksum', SON([(u'sha256', u'834fb545f765603e697ab016755f59afe48c22fb7a2e1411514a165b28ea36d8')])), (u'_id', u'8adc9f17-4e41-4264-8d4e-4cfba816ebea')])] Lookup batch repo-package RepoApi.get_file_checksum: {'foo1234': {'pyxf86config-0.3.37-8.fc14.x86_64.rpm': SON([(u'sha256', u'dab399b4d9d730cb0bd129745bfcbac32157c54f800e0f65718402cc75ff4b81')])}, 'foo123': {'pulp-cds-0.0.121-1.fc14.noarch.rpm': SON([(u'sha256', u'834fb545f765603e697ab016755f59afe48c22fb7a2e1411514a165b28ea36d8')])}}
moving back to assigned as per prads request prad yea looks like some one remove the check i had for invalid case..; if you remove foo it works..try the script now.. prad move it back to assigned and I'll add a fix for invalid case
commit 86a8ac1943ecc6d0f56a906893d96907bcc3628f Author: Pradeep Kilambi <pkilambi> Date: Tue Jan 18 10:38:31 2011 -0500 636564 - handle the case where package is not in the repo # ./test.py Lookup package checksum PackageApi.package_checksum: [SON([(u'checksum', SON([(u'sha256', u'834fb545f765603e697ab016755f59afe48c22fb7a2e1411514a165b28ea36d8')])), (u'_id', u'8adc9f17-4e41-4264-8d4e-4cfba816ebea')])] Lookup batch repo-package RepoApi.get_file_checksum: {'foo1234': {'foo': None, 'pyxf86config-0.3.37-8.fc14.x86_64.rpm': SON([(u'sha256', u'dab399b4d9d730cb0bd129745bfcbac32157c54f800e0f65718402cc75ff4b81')])}, 'foo123': {'pulp-cds-0.0.121-1.fc14.noarch.rpm': SON([(u'sha256', u'834fb545f765603e697ab016755f59afe48c22fb7a2e1411514a165b28ea36d8')])}} If package is missing checksum should be None now...
on_qa build 128
verified [root@preethi ~]# rpm -q pulp pulp-0.0.128-1.fc14.noarch [root@preethi ~]# pulp-admin package info --name=pyxf86config --repoid=foo1234 +------------------------------------------+ Package Information +------------------------------------------+ vendor: None name: pyxf86config repo_defined: False checksum: {'sha256': 'dab399b4d9d730cb0bd129745bfcbac32157c54f800e0f65718402cc75ff4b81'} download_url: https://preethi.usersys.redhat.com//pulp/repos/foo1234/pyxf86config-0.3.37-8.fc14.x86_64.rpm filename: pyxf86config-0.3.37-8.fc14.x86_64.rpm epoch: 0 version: 0.3.37 arch: x86_64 provides: ['pyxf86config(x86-64)', 'pyxf86config', 'ixf86configmodule.so()(64bit)'] _ns: packages release: 8.fc14 _id: 5f1ed442-ad2c-49bb-b3e4-222862ad32e7 requires: ['libc.so.6(GLIBC_2.2.5)(64bit)', 'libc.so.6()(64bit)', 'libpthread.so.0(GLIBC_2.2.5)(64bit)', 'libgmodule-2.0.so.0()(64bit)', 'libgthread-2.0.so.0()(64bit)', 'rtld(GNU_HASH)', 'libc.so.6(GLIBC_2.3.4)(64bit)', 'libpthread.so.0()(64bit)', 'libglib-2.0.so.0()(64bit)', 'libgobject-2.0.so.0()(64bit)', 'librt.so.1()(64bit)', 'libc.so.6(GLIBC_2.3)(64bit)', 'python(abi)'] id: 5f1ed442-ad2c-49bb-b3e4-222862ad32e7 description: Python wrappers for the X server config file library libxf86config. It is used to read and write X server configuration files.
Also ran the script [root@preethi ~]# cat test.py #! /usr/bin/python from pulp.server.db import connection connection.initialize() from pulp.server import auditing auditing.initialize() from pulp.server.api.package import PackageApi from pulp.server.api.repo import RepoApi p = PackageApi() print "Lookup package checksum PackageApi.package_checksum:\n" print p.package_checksum('pulp-cds-0.0.121-1.fc14.noarch.rpm') r = RepoApi() print "Lookup batch repo-package RepoApi.get_file_checksum:\n" print r.get_file_checksums({'foo123' : ['pulp-cds-0.0.121-1.fc14.noarch.rpm'], 'foo1234' : ['pyxf86config-0.3.37-8.fc14.x86_64.rpm', 'foo']}) print r.get_file_checksums({'foo123' : ['pulp-cds-0.0.121-1.fc14.noarch.rpm'], 'foo1234' : ['pyxf86config-0.3.37-8.fc14.x86_64.rpm']}) [root@preethi ~]# pulp-admin repo create --id=foo1234 --name=foo1234 Successfully created repository [ foo1234 ] [root@preethi ~]# pulp-admin repo create --id=foo123 --name=foo123 Successfully created repository [ foo123 ] [root@preethi ~]# pulp-admin repo upload --id=foo123 --dir=/root/repo-dir/ Successful uploaded [pulp-cds-0.0.121-1.fc14.noarch.rpm] to repo [ foo123 ] [root@preethi ~]# pulp-admin repo upload --id=foo1234 --dir=/root/repo1/ Successful uploaded [pyxf86config-0.3.37-8.fc14.x86_64.rpm] to repo [ foo1234 ] [root@preethi ~]# ./test.py Lookup package checksum PackageApi.package_checksum: [SON([(u'checksum', SON([(u'sha256', u'834fb545f765603e697ab016755f59afe48c22fb7a2e1411514a165b28ea36d8')])), (u'_id', u'8b43b384-1b39-42a7-b2e7-0c83d93b4efa')])] Lookup batch repo-package RepoApi.get_file_checksum: {'foo1234': {'foo': None, 'pyxf86config-0.3.37-8.fc14.x86_64.rpm': SON([(u'sha256', u'dab399b4d9d730cb0bd129745bfcbac32157c54f800e0f65718402cc75ff4b81')])}, 'foo123': {'pulp-cds-0.0.121-1.fc14.noarch.rpm': SON([(u'sha256', u'834fb545f765603e697ab016755f59afe48c22fb7a2e1411514a165b28ea36d8')])}} {'foo1234': {'pyxf86config-0.3.37-8.fc14.x86_64.rpm': SON([(u'sha256', u'dab399b4d9d730cb0bd129745bfcbac32157c54f800e0f65718402cc75ff4b81')])}, 'foo123': {'pulp-cds-0.0.121-1.fc14.noarch.rpm': SON([(u'sha256', u'834fb545f765603e697ab016755f59afe48c22fb7a2e1411514a165b28ea36d8')])}}
Closing with Community Release 15 pulp-0.0.223-4.