The current "beaker-import" command is quite low level - it operates on a single distribution at a time. One more practical model (that is still independent of the mechanism used to transfers trees to a lab) is to have a directory where new trees are saved, with a script that periodically checks the directory for new trees (or, if the transfer mechanism is a bit more sophisticated, may be triggered on demand when new trees are known to be available). Providing such a mechanism as part of the Beaker lab controller packages will give creators of new Beaker installations the choice between using our suggested directory layout for distro trees and our higher level import utility, or else using their own layout and calling beaker-import directly as needed.
I think we can just discard our internal imports scripts if we add the following to beaker-import: --scan (or --recursive?) Recursively scans for .composeinfo files and imports each one it finds. The positional arguments must be URLs pointing at the same storage, and at least one must be a filesystem path. For example: beaker-import --scan http://example.com/distros/ ftp://example.com/distros/ /net/example.com/distros/ Once a particular path has been scanned it is added to an internal persistent state (/var/cache) so that it is not imported again the next time --scan is used. --ignore-scan-state Used with --scan. Import all found distros regardless of existing import state. --ignore-unimportable (or --ignore-uninstallable?) If a .composeinfo or .treeinfo is missing data needed for import, silently skip it instead of printing an error. (This is needed for layered products which may contain .composeinfo/.treeinfo but which are not installable, i.e. do not have netboot images.)
I like it. A few nitpicks/questions - I prefer "--recursive" to "--scan" - perhaps "/var/beaker/import-cache" for the cache location? - I suggest "--reimport" for the "override existing cache entries" option (since this makes sense as a standalone option, not just in combination with --scan) - I suggest "--skip-broken" for the option to silently ignore incomplete info files
I think we would need ways to limit what a recursive scan of the directories would find. I'm guessing we would want a way to include/exclude variants, arch, and families.
(In reply to Nick Coghlan from comment #4) > - perhaps "/var/beaker/import-cache" for the cache location? FHS does not permit this (non-standard subdir of /var). It could be /var/cache/beaker-import-status or /var/cache/beaker/import-status or some other variation. > - I suggest "--reimport" for the "override existing cache entries" option > (since this makes sense as a standalone option, not just in combination with > --scan) I was thinking that the import status cache would be ignored anyway when not using --scan. Or is that too inconsistent? > - I suggest "--skip-broken" for the option to silently ignore incomplete > info files The only reason not to use the word "broken" or similar is that the .composeinfo might be valid (as far as Release Engineering-type folks are concerned) but it's for a product which lacks netboot images and is therefore not installable and cannot sensible be imported into Beaker. beaker-import should distinguish between actual errors (.composeinfo has syntax errors) vs. uninstallable trees (.composeinfo lacks suitable image definitions) since the former is a real error that the admin should hear about, whereas the latter can just be silently skipped.
* /var/cache/beaker/import-status sounds good as an FHS compliant dir name that still indicates the cache belongs to beaker. * I think it's likely to be more user friendly to always use the cache, with a flag to force the tree to be reimported anyway. The fact the alternative becomes tricky to name and document hints that it's an inherently confusing approach. * Perhaps the reporting of trees without installable images can just be off by default and controlled by a verbosity flag?
(In reply to Nick Coghlan from comment #10) > * /var/cache/beaker/import-status sounds good as an FHS compliant dir name > that still indicates the cache belongs to beaker. Oh yeah, in comment 3 when I said /var/cache I meant a suitably named sub-dir of it, I wasn't suggesting we take over all of /var/cache :-) > * I think it's likely to be more user friendly to always use the cache, with > a flag to force the tree to be reimported anyway. The fact the alternative > becomes tricky to name and document hints that it's an inherently confusing > approach. Agreed. > * Perhaps the reporting of trees without installable images can just be off > by default and controlled by a verbosity flag? The issue with this is when you are importing a single compose/tree. In that case, the user really needs to know that the tree they are trying to import is not usable. The fact that they have point beaker-import at it means they clearly think it *is* usable, so if beaker-import silently did nothing it would be quite confusing. However when doing a recursive scan we really want to be able to just skip these uninstallable trees. Right now I get cron spammed several times each night (so do you, if you don't filter it :-P) due to our Perl script trying to import uninstallable layered products into Beaker, just because they contain .composeinfo.
Perhaps "--ignore-skipped" as the flag, then? And yes, I have plenty of spam in my cron job folder courtesy of that Perl script :)
https://fedorahosted.org/fedora-qa/ticket/437 suggests this may be relevant for beaker.fedoraproject.org as well. Dan, did you have a work-in-progress patch that could be put up on Gerrit?
(In reply to Nick Coghlan from comment #16) > Dan, did you have a work-in-progress patch that could be put up on Gerrit? Sure, this is where I got up to before I went on PTO last month: http://gerrit.beaker-project.org/2528 Leaving this bug at ASSIGNED because the patch still needs more work.
At this point it is more likely that we will remove Beaker's distro library functionality in favour of an external service (maybe PDC) although we have no firm plans in that regard yet.
Closing this as will not do, this feature is not impacting the RHEL 9 support that we are focusing on in the final LTS 28 version. If you think that this does have an impact on RHEL 9 support. Please give the business justification and reopen. If you have any question please contact Theo tapazogl.