Bug 1412488

Summary:	beaker-expire-distros cannot handle https:// URLs
Product:	[Retired] Beaker	Reporter:	Dan Callaghan <dcallagh>
Component:	general	Assignee:	Anwesha Chatterjee <achatter>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Roman Joost <rjoost>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	23	CC:	achatter, dcallagh, mjia, rjoost
Target Milestone:	25.0	Keywords:	Patch
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-03-19 04:19:06 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Dan Callaghan 2017-01-12 07:16:33 UTC

Version-Release number of selected component (if applicable):
23.3

How reproducible:
somewhat easily

Steps to Reproduce:
1. beaker-import https://dl.fedoraproject.org/pub/fedora-secondary/releases/25/Server/aarch64/os/
2. Run beaker-expire-distros (or let the cron job run)

Actual results:
Traceback (most recent call last):
  File "/usr/bin/beaker-expire-distros", line 9, in <module>
    load_entry_point('beaker-lab-controller==23.4.git.165.6a7a61b', 'console_scripts', 'beaker-expire-distros')()
  File "/usr/lib/python2.6/site-packages/bkr/labcontroller/expire_distros.py", line 110, in main
    check_all_trees(options.ignore_errors)
  File "/usr/lib/python2.6/site-packages/bkr/labcontroller/expire_distros.py", line 76, in check_all_trees
    if check_url(url):
  File "/usr/lib/python2.6/site-packages/bkr/labcontroller/expire_distros.py", line 66, in check_url
    raise ValueError('Unrecognised URL scheme %s for tree %s' % (scheme, url))
ValueError: Unrecognised URL scheme https for tree https://dl.fedoraproject.org/pub/fedora-secondary/releases/25/Server/aarch64/os/

Expected results:
Should be able to check https:// URLs the same way it can do http:// URLs.

Comment 1 Dan Callaghan 2017-01-12 07:20:52 UTC

beaker-pxemenu also barfs if the only URL for the tree is https://, maybe fix this at the same time?

Fetching kernel images/pxeboot/vmlinuz for distro tree 6006
Traceback (most recent call last):
  File "/usr/bin/beaker-pxemenu", line 9, in <module>
    load_entry_point('beaker-lab-controller==23.4.git.165.6a7a61b', 'console_scripts', 'beaker-pxemenu')()
  File "/usr/lib/python2.6/site-packages/bkr/labcontroller/pxemenu.py", line 153, in main
    write_menus(opts.tftp_root, opts.tags, opts.xml_filter)
  File "/usr/lib/python2.6/site-packages/bkr/labcontroller/pxemenu.py", line 107, in write_menus
    distro_trees = _get_all_images(tftp_root, distro_trees)
  File "/usr/lib/python2.6/site-packages/bkr/labcontroller/pxemenu.py", line 61, in _get_all_images
    url, distro_tree['images'])
  File "/usr/lib/python2.6/site-packages/bkr/labcontroller/pxemenu.py", line 49, in _get_images
    siphon(urllib2.urlopen(image_url), dest)
  File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib64/python2.6/urllib2.py", line 383, in open
    protocol = req.get_type()
  File "/usr/lib64/python2.6/urllib2.py", line 244, in get_type
    raise ValueError, "unknown url type: %s" % self.__original
ValueError: unknown url type: images/pxeboot/vmlinuz

Comment 2 Anwesha Chatterjee 2018-01-24 05:45:17 UTC

Some more information about running beaker-expire-distros would be good, as I cant seem to reproduce this stack trace on lab-devel-02. Also

* there is no documentation for it
* beaker-expire_distros being an rpm - its hard to debug
* expire_distros.py isnt straightforward to run locally on the dev server. 

importing works fine.

Comment 3 Roman Joost 2018-01-25 06:40:42 UTC

I can reproduce the problem. Will go through this on Monday with Anwesha.

Comment 4 Anwesha Chatterjee 2018-01-30 07:13:34 UTC

https://gerrit.beaker-project.org/#/c/5984/

This patch was surprisingly hard to write a test case for. I have created a draft patch here https://gerrit.beaker-project.org/#/c/5982/ where I attempted to create a https server to test https urls in expire_distros. Appreciate feedback on the attempt.

Comment 5 Dan Callaghan 2018-01-30 09:18:25 UTC

Personally I think it would be acceptable to not have an automated test covering this, and just test it by hand, due to the difficult in messing around with SSL certs for the fake mirror server.

Comment 9 Roman Joost 2018-03-19 04:19:06 UTC

Beaker 25.0 has been released.

Release notes are available upstream: https://beaker-project.org/docs/whats-new/release-25.html