Bug 666435 - cobbler buildiso incorrectly listed in Reference Guide - Was: cobbler get-loaders ignores proxy setting
cobbler buildiso incorrectly listed in Reference Guide - Was: cobbler get-loa...
Product: Red Hat Satellite 5
Classification: Red Hat
Component: Docs Deployment Guide (Show other bugs)
All All
medium Severity medium
: ---
: ---
Assigned To: Lana Brindley
Martin Minar
: Documentation, Reopened
Depends On:
Blocks: 644720 sat54-docs
  Show dependency treegraph
Reported: 2010-12-30 13:20 EST by Marcelo Moreira de Mello
Modified: 2016-07-03 20:55 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2011-06-23 17:27:22 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Patch Proposed (1.78 KB, patch)
2010-12-30 17:12 EST, Marcelo Moreira de Mello
no flags Details | Diff
Patch for /etc/init.d/cobblerd (322 bytes, patch)
2011-01-13 14:30 EST, Marcelo Moreira de Mello
no flags Details | Diff

External Trackers
Tracker ID Priority Status Summary Last Updated
FedoraHosted Cobbler 572 None None None Never
Red Hat Knowledge Base (Solution) 259863 None None None Never
Red Hat Knowledge Base (Solution) 28978 None None None Never

  None (edit)
Description Marcelo Moreira de Mello 2010-12-30 13:20:27 EST
Description of problem:

When running "cobbler get-loaders" using http_proxy, cobbler does not recognized configuration. 

[root@server ~]# env | grep -ie http 

[root@server ~]# cobbler get-loaders
task started: 2010-12-30_140238_get_loaders
task started (id=Download Bootloader Content, time=Thu Dec 30 14:02:38 2010)
downloading content required to netboot all arches
downloading http://mdehaan.fedorapeople.org/loaders/README to /var/lib/cobbler/loaders/README
Exception occured: urlgrabber.grabber.URLGrabError
Exception value: [Errno 4] IOError: <urlopen error (111, 'Connection refused')>
Exception Info:
  File "/usr/lib/python2.4/site-packages/cobbler/remote.py", line 95, in run
    rc = self._run(self)
   File "/usr/lib/python2.4/site-packages/cobbler/remote.py", line 179, in runner
    return self.remote.api.dlcontent(self.options.get("force",False), self.logger)
   File "/usr/lib/python2.4/site-packages/cobbler/api.py", line 582, in dlcontent
    return grabber.run(force)
   File "/usr/lib/python2.4/site-packages/cobbler/action_dlcontent.py", line 71, in run
   File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 602, in urlgrab
    return default_grabber.urlgrab(url, filename, **kwargs)
   File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 934, in urlgrab
    return self._retry(opts, retryfunc, url, filename)
   File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 852, in _retry
    r = apply(func, (opts,) + args, {})
   File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 920, in retryfunc
    fo = URLGrabberFileObject(url, filename, opts)
   File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 1008, in __init__
   File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 1091, in _do_open
    fo, hdr = self._make_request(req, opener)
   File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 1220, in _make_request
    raise URLGrabError(4, _('IOError: %s') % (e, ))


Version-Release number of selected component (if applicable):
Red Hat Network Satellite  5.4

How reproducible:

Steps to Reproduce:
1. Export HTTP_PROXY variable
2. run cobbler get-loaders
Actual results:
Does not recognize http_proxy configuration 

Expected results:
Recognize http_proxy configuration
Comment 1 Marcelo Moreira de Mello 2010-12-30 13:23:04 EST
### See also: 
Comment 2 Marcelo Moreira de Mello 2010-12-30 17:12:27 EST
Created attachment 471221 [details]
Patch Proposed

Howdy team, 

I wrote a patch for honor the environment variables. 

Follow the test procedure: 

  a) upgrade package
[root@server ~]# rpm -Uvh cobbler-2.0.7-5.el5sat.TEST.00387446.3.noarch.rpm 
Preparing...                ########################################### [100%]
   1:cobbler                ########################################### [100%]
Stopping cobbler daemon: [  OK  ]
Starting cobbler daemon: SERVING!
[  OK  ]
[root@server ~]# 
  b) showing env var

[root@server ~]# env | grep http

  c) no files in directory 

[root@server ~]# tree /var/lib/cobbler/loaders/

0 directories, 0 files

  d) close output on iptables to ensure traffic uses proxy

[root@server ~]# iptables -A OUTPUT -p tcp --dport 80 -o eth0 -j LOG
[root@server ~]# iptables -A OUTPUT -p tcp --dport 443 -o eth0 -j LOG
[root@server ~]# iptables -A OUTPUT -p tcp --dport 80 -o eth0 -j REJECT
[root@server ~]# iptables -A OUTPUT -p tcp --dport 443 -o eth0 -j REJECT
[root@server ~]# iptables -L -n -v 
Chain INPUT (policy ACCEPT 123M packets, 81G bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 105M packets, 50G bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 LOG        tcp  --  *      eth0             tcp dpt:80 LOG flags 0 level 4 
   87  5205 LOG        tcp  --  *      eth0             tcp dpt:443 LOG flags 0 level 4 
    0     0 REJECT     tcp  --  *      eth0             tcp dpt:80 reject-with icmp-port-unreachable 
   17  2264 REJECT     tcp  --  *      eth0             tcp dpt:443 reject-with icmp-port-unreachable 

   e) get loaders

[root@server ~]# cobbler get-loaders
task started: 2010-12-30_180723_get_loaders
task started (id=Download Bootloader Content, time=Thu Dec 30 18:07:23 2010)
downloading http://mdehaan.fedorapeople.org/loaders/README to /var/lib/cobbler/loaders/README
downloading http://mdehaan.fedorapeople.org/loaders/COPYING.elilo to /var/lib/cobbler/loaders/COPYING.elilo
downloading http://mdehaan.fedorapeople.org/loaders/COPYING.yaboot to /var/lib/cobbler/loaders/COPYING.yaboot
downloading http://mdehaan.fedorapeople.org/loaders/COPYING.syslinux to /var/lib/cobbler/loaders/COPYING.syslinux
downloading http://mdehaan.fedorapeople.org/loaders/elilo-3.8-ia64.efi to /var/lib/cobbler/loaders/elilo-ia64.efi
downloading http://mdehaan.fedorapeople.org/loaders/yaboot-1.3.14-12 to /var/lib/cobbler/loaders/yaboot
downloading http://mdehaan.fedorapeople.org/loaders/pxelinux.0-3.61 to /var/lib/cobbler/loaders/pxelinux.0
downloading http://mdehaan.fedorapeople.org/loaders/menu.c32-3.61 to /var/lib/cobbler/loaders/menu.c32
   f) files downloaded
[root@server ~]# tree -upgt /var/lib/cobbler/loaders/
|-- [-rw-r--r-- root     root    ]  yaboot-1.3.14-12
|-- [-rw-r--r-- root     root    ]  pxelinux.0-3.61
|-- [-rw-r--r-- root     root    ]  menu.c32-3.61
|-- [-rw-r--r-- root     root    ]  COPYING.syslinux
|-- [-rw-r--r-- root     root    ]  README
|-- [-rw-r--r-- root     root    ]  elilo-3.8-ia64.efi
|-- [-rw-r--r-- root     root    ]  COPYING.yaboot
`-- [-rw-r--r-- root     root    ]  COPYING.elilo

0 directories, 8 files
[root@server ~]# 

Proxy Output: 
  $ tail -f /var/log/squid/access.log

1293760629.586     61 TCP_MISS/200 1532 GET http://mdehaan.fedorapeople.org/loaders/README - DIRECT/ text/plain
1293760629.645     53 TCP_MISS/200 1115 GET http://mdehaan.fedorapeople.org/loaders/COPYING.elilo - DIRECT/ text/plain
1293760629.710     53 TCP_MISS/200 1110 GET http://mdehaan.fedorapeople.org/loaders/COPYING.yaboot - DIRECT/ text/plain
1293760629.829    113 TCP_MISS/200 18494 GET http://mdehaan.fedorapeople.org/loaders/COPYING.syslinux - DIRECT/ text/plain
1293760630.133    302 TCP_MISS/200 356982 GET http://mdehaan.fedorapeople.org/loaders/elilo-3.8-ia64.efi - DIRECT/ text/plain
1293760630.414    278 TCP_MISS/200 198681 GET http://mdehaan.fedorapeople.org/loaders/yaboot-1.3.14-12 - DIRECT/ text/plain
1293760630.557    139 TCP_MISS/200 15203 GET http://mdehaan.fedorapeople.org/loaders/pxelinux.0-3.61 - DIRECT/ text/plain
1293760630.753    192 TCP_MISS/200 39675 GET http://mdehaan.fedorapeople.org/loaders/menu.c32-3.61 - DIRECT/ text/plain
Comment 3 Marcelo Moreira de Mello 2011-01-13 14:30:27 EST
Created attachment 473400 [details]
Patch for /etc/init.d/cobblerd


I made some changes on our spec file, which includes an option to read the /etc/sysconfig/cobblerd to set the http_proxy variable for read. This way guarantee that variables will be loaded when cobblerd  service will be load.  

$ cat /etc/sysconfig/cobblerd 
export HTTP_PROXY="http://proxy_server:3128"

$ service cobblerd restart

$ cobbler get-loaders 

Please, add this on the next cobblerd package for Red Hat. 

Marcelo Moreira de Mello
Comment 4 Marcelo Moreira de Mello 2011-01-13 14:35:15 EST

Just to share that patch were committed in upstream. 

$> git show 0218888f9e72a6ffb358003f19686b040d7fd02a
commit 0218888f9e72a6ffb358003f19686b040d7fd02a
Author: Scott Henson <shenson@redhat.com>
Date:   Wed Jan 12 12:08:22 2011 -0500

    Allow dlcontent to use proxy environment variables

diff --git a/cobbler/action_dlcontent.py b/cobbler/action_dlcontent.py
index b3adae0..d73f114 100644
--- a/cobbler/action_dlcontent.py
+++ b/cobbler/action_dlcontent.py
@@ -62,13 +62,33 @@ class ContentDownloader:
           ( "%s/grub-0.97-x86_64.efi" % content_server, "%s/grub-x86_64.efi" % dest),
-       self.logger.info("downloading content required to netboot all arches")
-       for f in files:
-          src = f[0]
-          dst = f[1]
-          if os.path.exists(dst) and not force:
-             self.logger.info("path %s already exists, not overwriting existing content, use --force if you wish to update" % dst)
-             continue
+       proxies = {} 
+       for var in os.environ.keys():
+          if (var.lower() == "http_proxy"):
+             proxies['http'] = os.environ[var]
+          if (var.lower() == "ftp_proxy"):
+             proxies['ftp'] = os.environ[var]
+       if (len(proxies) > 0):
+          for f in files:
+             src = f[0]
+             dst = f[1]
+             if os.path.exists(dst) and not force:
+                self.logger.info("path %s already exists, not overwriting existing content, use --force if you wish to update" % dst)
+                continue
+             os.chdir(dest)
+             self.logger.info("downloading %s to %s" % (src,dst))
+             urlgrabber.grabber.urlgrab(src, proxies=proxies)  
+       else:
+          self.logger.info("downloading content required to netboot all arches")
+          for f in files:
+             src = f[0]
+             dst = f[1]
+             if os.path.exists(dst) and not force:
+                self.logger.info("path %s already exists, not overwriting existing content, use --force if you wish to update" % dst)
+                continue
           self.logger.info("downloading %s to %s" % (src,dst))
Comment 9 Marcelo Moreira de Mello 2011-01-17 05:48:51 EST

Since the patch already were committed in upstream, for futures cobbler packages shipped by Red Hat honour the fix, we probably will just need to add in our SPEC the https://bugzilla.redhat.com/attachment.cgi?id=473400 and a new NOTE in our documentation that proxy is supported by doing a configuration in /etc/sysconfig/cobblerd. 

Best Regards, 
Marcelo Moreira de Mello
Comment 12 Clifford Perry 2011-01-19 12:32:22 EST
Lana - within changes made for bug 492961 - which was being used to track corrections to errors in the cobbler documentation, it seems we added a section for 'cobbler buildiso' usage by mistake (which was a proposed addition). 

This has caused this bug - bug 666435 & bug 667924 to be reported against Satellite 5.4.0. 

As noted within 506485 we have not spent time for development or QE in testing this functionality to flesh it out, as also shown by these two new bug reports. 

I am converting this into a documentation bug for Satellite 5.4.1 release cycle. Can we please use it to track the removal of cobbler buildiso from our reference guide. 

I'm made this comment public to allow GSS and others to easily find this. I've also changed the subject of the bug to reflect the change. 

Comment 13 Gilles Dubreuil 2011-01-31 07:30:53 EST
This could not be a documentation bug as this feature exists and is supported in Satellite 5.3.

As a matter of fact we have customers using the feature and not supporting will mean breaking their current build scenarios.

That feature works very well and comes straight for Spacewalk.

It seems that few things were moved around between two versions of Spacewalk which occured between Satellite 5.3 to 5.4.

The work around I found to fix it in Satellite 5.4 is:

Used /var/lib/cobbler/menu.c32 from RHN Satellite 5.3 and copied into RHN Satellite 5.4 as /var/lib/cobbler/loaders/menu.c32

After that Cobbler build iso works:

# cobbler buildiso
task started: 2010-11-02_182138_buildiso
task started (id=Build Iso, time=Tue Nov 2 18:21:38 2010)
using/creating tempdir: /buildiso
building tree for isolinux
copying miscellaneous files
copying kernels and initrds for profiles
generating a isolinux.cfg
generating profile list
done writing config
running: mkisofs -o /root/generated.iso -r -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -V Cobbler\ Install -R -J -T /buildiso
ISO build complete
You may wish to delete: /buildiso
The output file is: /root/generated.iso


Comment 15 Aurelien Gouny 2011-02-01 22:06:58 EST
What happened to "we support what we ship" ?

I've just hit the exact same buildiso issue at a major customer & used Gilles' fix to be able to correct this after a Satellite upgrade, the customer has been using 'cobbler buildiso' heavily since 5.3 because of network restrictions due to security, it is my opinion that we can't just remove functions that we've shipped previously.

Thanks Gilles.
Aurelien (GPS Consultant - Sydney, AU)
Comment 16 Aurelien Gouny 2011-02-01 22:15:24 EST
To be exact and to detail further:
- compared to Gilles, I didn't have any /var/lib/cobbler/menu.c32
- tried 'cobbler get-loaders' but as the system is behind a proxy -> bug
- copied /usr/lib/syslinux/menu.c32 to /var/lib/cobbler/loaders/menu.c32 -> ok

'cobbler buildiso' now works fine.
Aurelien (GPS Consultant - Sydney, AU)
Comment 17 Gilles Dubreuil 2011-02-01 23:27:28 EST
It's more than just a doco mistake. Because of the impact on customers

The functionality was in 5.3 and documented and still is in 5.4 and documented.
Now if one cannot build servers with PXE Boot than we have to provide a way to build ISOs.


Comment 18 Lana Brindley 2011-03-21 23:26:37 EDT
In the Kickstart chapter in the new User Guide, I have included the following procedure:

<procedure id="proc-Getting_Started_Guide-Kickstarting_from_Bare_Metal_-Cobbler_Boot_ISO">
<title>Cobbler Boot ISO</title>
		The Cobbler boot ISO is a disk image that can be built on a &SAT; server and burned to a CD or DVD. It can then be used to boot a bare metal system.
		To create a boot ISO, log in to the &SAT; server as root and run <command>cobbler buildiso</command>. The ISO will contain all kernel and initrd images stored in the &SAT;, along with all associated kernel argument settings.
		Kickstart files will be sourced remotely, which means that changes to the kickstart templates can be made without having to re-burn the disk. However, if you create a new kickstart profile and want to use it with the Cobbler boot ISO, you will need to create a new disk.
		Insert the Cobbler boot ISO disk. A menu of available kickstarts will be displayed. Select the appropriate kickstart and the system will start to install automatically. <remark>A screenshot here would be good. LKB</remark>
		Any time you add a kickstart within Satellite, you will need to recreate the ISO and re-burn it to an optical disk.

For the purposes of fixing this bug, I have commented the procedure out. If it is decided later on to have this procedure visible, please raise a new bug.

Comment 21 Pavel Novotny 2011-04-21 08:24:53 EDT
Taking this bug for verification.
Comment 25 Lana Brindley 2011-05-05 20:18:56 EDT
This book has now been dropped to translation (RT#75265).
No further updates can be accepted. Please raise a new bug for any changes.
Comment 26 Lana Brindley 2011-06-16 18:14:06 EDT
5.4.1 Satellite books are now available on docs.redhat.com. Please raise a new bug for any issues.

Comment 27 Lana Brindley 2011-06-21 23:51:39 EDT
Reopening, as Cliff found some more instances.
Comment 28 Lana Brindley 2011-06-22 00:28:53 EDT
Found and fixed several instances in 5.3 and 5.4 Deployment Guides.

Comment 29 Lana Brindley 2011-06-22 00:32:13 EDT
Requested respin in RT#114048


Note You need to log in before you can comment on or make changes to this bug.