Bug 1232342 - hawkey only works with XML repo metadata and not with sqlite databases
Summary: hawkey only works with XML repo metadata and not with sqlite databases
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: Fedora
Classification: Fedora
Component: hawkey
Version: 23
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Packaging Maintenance Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-06-16 14:14 UTC by Yavor Atanasov
Modified: 2015-07-29 10:57 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-07-29 10:50:39 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 850896 0 medium CLOSED [rfe] [metadata] smarter metadata format for quicker sync (delta/per-package/...) 2021-02-22 00:41:40 UTC

Internal Links: 850896

Description Yavor Atanasov 2015-06-16 14:14:32 UTC
Hey guys,

Currently, to initiate a hawkey Repo object, we need to provide the different types of metadata (primary, filelists etc.) in the form of xml files - http://hawkey.readthedocs.org/en/latest/reference-py-repo.html

Are there plans to have hawkey work with the sqlite databases as well or it is a conscious decision to ignore those in favour of xml.

Thank you
Yavor

Comment 1 Radek Holy 2015-06-16 15:14:16 UTC
Hi, I'd say that if we/you find a valid use case, we can consider adding the support.

Comment 2 Yavor Atanasov 2015-06-17 10:36:49 UTC
Hi Radek,

To give you a bit of context, my team is part of BBC's online platform teams. More specifically we create the services and tools that form the deployment pipeline for all of BBC's "cloud" services. Part of that ecosystem are a couple of services - one that bakes virtual machine images based on a set of repositories, packages and configuration, and another which is essentially a yum repository as a service - it allows our teams to create and maintain isolated repos. The repo service generates the yum metadata for a given repository on the fly when requested. Generating the metadata in the form of sqlite files is very fast for us as the data model behind the service mirrors closely the schema required for sqlite files (all we really do is copy required tables into sqlite, compress it and serve it). We do not currently generate XML files as yum works happily with sqlite.

We are migrating the bakery services to use hawkey and librepo. We want to use some of that goodness to speed up the baking process and do smarter things with pulling repo metadata, packages, installation etc.

It has all been great before we hit the problem I alluded to above - hawkey cannot currently work with our repository service as it requires XML files. You can see why it would be beneficial for us if hawkey could read the metadata from sqlite instead.

Out of curiosity what is the reason for choosing XML over sqlite (if there was a particular reason)?

Anyway, that's our use case.
Yavor

Comment 3 Radek Holy 2015-06-17 10:55:28 UTC
Thanks for the use case. Makes a lot of sense.

BTW, one problem that annoys me all the time is that the repodata format is not documented. So far, I looked at "createrepo" when I wanted to know something about the format. Using this as the documentation of the format, one can say that the XML files are mandatory since it's not possible to create sqlite-only repository using "createrepo". And if they are mandatory, we don't need the sqlite files.

Anyway, I guess that the main reason is that hawkey is a wrapper around libsolv and libsolv doesn't seem to support sqlite files.

Comment 4 Yavor Atanasov 2015-06-18 13:31:21 UTC
Would Red Hat be open to additions to libsolv so that works with sqlite files as well? I just want to rule out a possibility in which RH wants to standardise the poorly documented repo metadata formats and do that in favour of XML trying to deprecate sqlite.

Yavor

Comment 5 Radek Holy 2015-06-19 08:55:52 UTC
I believe that this is more a question for the libsolv upstream: https://github.com/openSUSE/libsolv

Comment 6 Yavor Atanasov 2015-06-19 09:27:27 UTC
Ah yes, I missed the fact that their github repo accepts issues to be raised (as opposed to the hawkey repo). Will raise an issue there.

Thank you Radek.

Comment 7 Jan Kurik 2015-07-15 13:59:40 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 23 development cycle.
Changing version to '23'.

(As we did not run this process for some time, it could affect also pre-Fedora 23 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 23 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora23

Comment 8 Honza Silhan 2015-07-29 10:50:39 UTC
see issue in libsolv project: https://github.com/openSUSE/libsolv/issues/92

solv files would be more preferred. If you need exactly sqlite format, please, post your use case.

Comment 9 Honza Silhan 2015-07-29 10:57:27 UTC
* I mean the use case of other users wanting this feature.


Note You need to log in before you can comment on or make changes to this bug.