Bug 1412488 - beaker-expire-distros cannot handle https:// URLs
Summary: beaker-expire-distros cannot handle https:// URLs
Alias: None
Product: Beaker
Classification: Community
Component: general
Version: 23
Hardware: Unspecified
OS: Unspecified
Target Milestone: 25.0
Assignee: Anwesha Chatterjee
QA Contact: Roman Joost
Depends On:
TreeView+ depends on / blocked
Reported: 2017-01-12 07:16 UTC by Dan Callaghan
Modified: 2018-03-19 04:19 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2018-03-19 04:19:06 UTC

Attachments (Terms of Use)

Description Dan Callaghan 2017-01-12 07:16:33 UTC
Version-Release number of selected component (if applicable):

How reproducible:
somewhat easily

Steps to Reproduce:
1. beaker-import https://dl.fedoraproject.org/pub/fedora-secondary/releases/25/Server/aarch64/os/
2. Run beaker-expire-distros (or let the cron job run)

Actual results:
Traceback (most recent call last):
  File "/usr/bin/beaker-expire-distros", line 9, in <module>
    load_entry_point('beaker-lab-controller==23.4.git.165.6a7a61b', 'console_scripts', 'beaker-expire-distros')()
  File "/usr/lib/python2.6/site-packages/bkr/labcontroller/expire_distros.py", line 110, in main
  File "/usr/lib/python2.6/site-packages/bkr/labcontroller/expire_distros.py", line 76, in check_all_trees
    if check_url(url):
  File "/usr/lib/python2.6/site-packages/bkr/labcontroller/expire_distros.py", line 66, in check_url
    raise ValueError('Unrecognised URL scheme %s for tree %s' % (scheme, url))
ValueError: Unrecognised URL scheme https for tree https://dl.fedoraproject.org/pub/fedora-secondary/releases/25/Server/aarch64/os/

Expected results:
Should be able to check https:// URLs the same way it can do http:// URLs.

Comment 1 Dan Callaghan 2017-01-12 07:20:52 UTC
beaker-pxemenu also barfs if the only URL for the tree is https://, maybe fix this at the same time?

Fetching kernel images/pxeboot/vmlinuz for distro tree 6006
Traceback (most recent call last):
  File "/usr/bin/beaker-pxemenu", line 9, in <module>
    load_entry_point('beaker-lab-controller==23.4.git.165.6a7a61b', 'console_scripts', 'beaker-pxemenu')()
  File "/usr/lib/python2.6/site-packages/bkr/labcontroller/pxemenu.py", line 153, in main
    write_menus(opts.tftp_root, opts.tags, opts.xml_filter)
  File "/usr/lib/python2.6/site-packages/bkr/labcontroller/pxemenu.py", line 107, in write_menus
    distro_trees = _get_all_images(tftp_root, distro_trees)
  File "/usr/lib/python2.6/site-packages/bkr/labcontroller/pxemenu.py", line 61, in _get_all_images
    url, distro_tree['images'])
  File "/usr/lib/python2.6/site-packages/bkr/labcontroller/pxemenu.py", line 49, in _get_images
    siphon(urllib2.urlopen(image_url), dest)
  File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib64/python2.6/urllib2.py", line 383, in open
    protocol = req.get_type()
  File "/usr/lib64/python2.6/urllib2.py", line 244, in get_type
    raise ValueError, "unknown url type: %s" % self.__original
ValueError: unknown url type: images/pxeboot/vmlinuz

Comment 2 Anwesha Chatterjee 2018-01-24 05:45:17 UTC
Some more information about running beaker-expire-distros would be good, as I cant seem to reproduce this stack trace on lab-devel-02. Also

* there is no documentation for it
* beaker-expire_distros being an rpm - its hard to debug
* expire_distros.py isnt straightforward to run locally on the dev server. 

importing works fine.

Comment 3 Roman Joost 2018-01-25 06:40:42 UTC
I can reproduce the problem. Will go through this on Monday with Anwesha.

Comment 4 Anwesha Chatterjee 2018-01-30 07:13:34 UTC

This patch was surprisingly hard to write a test case for. I have created a draft patch here https://gerrit.beaker-project.org/#/c/5982/ where I attempted to create a https server to test https urls in expire_distros. Appreciate feedback on the attempt.

Comment 5 Dan Callaghan 2018-01-30 09:18:25 UTC
Personally I think it would be acceptable to not have an automated test covering this, and just test it by hand, due to the difficult in messing around with SSL certs for the fake mirror server.

Comment 9 Roman Joost 2018-03-19 04:19:06 UTC
Beaker 25.0 has been released.

Release notes are available upstream: https://beaker-project.org/docs/whats-new/release-25.html

Note You need to log in before you can comment on or make changes to this bug.