Bug 666435 - cobbler buildiso incorrectly listed in Reference Guide - Was: cobbler get-loaders ignores proxy setting
cobbler buildiso incorrectly listed in Reference Guide - Was: cobbler get-loa...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Satellite 5
Classification: Red Hat
Component: Docs Deployment Guide (Show other bugs)
540
All All
medium Severity medium
: ---
: ---
Assigned To: Lana Brindley
Martin Minar
: Documentation, Reopened
Depends On:
Blocks: 644720 sat54-docs
  Show dependency treegraph
 
Reported: 2010-12-30 13:20 EST by Marcelo Moreira de Mello
Modified: 2016-07-03 20:55 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-06-23 17:27:22 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Patch Proposed (1.78 KB, patch)
2010-12-30 17:12 EST, Marcelo Moreira de Mello
no flags Details | Diff
Patch for /etc/init.d/cobblerd (322 bytes, patch)
2011-01-13 14:30 EST, Marcelo Moreira de Mello
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
FedoraHosted Cobbler 572 None None None Never
Red Hat Knowledge Base (Solution) 259863 None None None Never
Red Hat Knowledge Base (Solution) 28978 None None None Never

  None (edit)
Description Marcelo Moreira de Mello 2010-12-30 13:20:27 EST
Description of problem:

When running "cobbler get-loaders" using http_proxy, cobbler does not recognized configuration. 


[root@server ~]# env | grep -ie http 
http_proxy=http://proxy-server.example.com:3128
FTP_PROXY=http://proxy-server.example.com:3128
ftp_proxy=http://proxy-server.example.com:3128
HTTP_PROXY=http://proxy-server.example.com:3128

[root@server ~]# cobbler get-loaders
task started: 2010-12-30_140238_get_loaders
task started (id=Download Bootloader Content, time=Thu Dec 30 14:02:38 2010)
downloading content required to netboot all arches
downloading http://mdehaan.fedorapeople.org/loaders/README to /var/lib/cobbler/loaders/README
Exception occured: urlgrabber.grabber.URLGrabError
Exception value: [Errno 4] IOError: <urlopen error (111, 'Connection refused')>
Exception Info:
  File "/usr/lib/python2.4/site-packages/cobbler/remote.py", line 95, in run
    rc = self._run(self)
   File "/usr/lib/python2.4/site-packages/cobbler/remote.py", line 179, in runner
    return self.remote.api.dlcontent(self.options.get("force",False), self.logger)
   File "/usr/lib/python2.4/site-packages/cobbler/api.py", line 582, in dlcontent
    return grabber.run(force)
   File "/usr/lib/python2.4/site-packages/cobbler/action_dlcontent.py", line 71, in run
    urlgrabber.urlgrab(src,dst)
   File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 602, in urlgrab
    return default_grabber.urlgrab(url, filename, **kwargs)
   File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 934, in urlgrab
    return self._retry(opts, retryfunc, url, filename)
   File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 852, in _retry
    r = apply(func, (opts,) + args, {})
   File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 920, in retryfunc
    fo = URLGrabberFileObject(url, filename, opts)
   File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 1008, in __init__
    self._do_open()
   File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 1091, in _do_open
    fo, hdr = self._make_request(req, opener)
   File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 1220, in _make_request
    raise URLGrabError(4, _('IOError: %s') % (e, ))

!!! TASK FAILED !!!

Version-Release number of selected component (if applicable):
cobbler-2.0.7-5.el5sat
Red Hat Network Satellite  5.4

How reproducible:
100%

Steps to Reproduce:
1. Export HTTP_PROXY variable
2. run cobbler get-loaders
  
Actual results:
Does not recognize http_proxy configuration 

Expected results:
Recognize http_proxy configuration
Comment 1 Marcelo Moreira de Mello 2010-12-30 13:23:04 EST
### See also: 
     https://fedorahosted.org/cobbler/ticket/572
Comment 2 Marcelo Moreira de Mello 2010-12-30 17:12:27 EST
Created attachment 471221 [details]
Patch Proposed

Howdy team, 

I wrote a patch for honor the environment variables. 

Follow the test procedure: 

  a) upgrade package
[root@server ~]# rpm -Uvh cobbler-2.0.7-5.el5sat.TEST.00387446.3.noarch.rpm 
Preparing...                ########################################### [100%]
   1:cobbler                ########################################### [100%]
Stopping cobbler daemon: [  OK  ]
Starting cobbler daemon: SERVING!
[  OK  ]
[root@server ~]# 
  
  b) showing env var

[root@server ~]# env | grep http
http_proxy=http://proxy-server.example.com:3128

  c) no files in directory 

[root@server ~]# tree /var/lib/cobbler/loaders/
/var/lib/cobbler/loaders/

0 directories, 0 files

  d) close output on iptables to ensure traffic uses proxy

[root@server ~]# iptables -A OUTPUT -p tcp --dport 80 -o eth0 -j LOG
[root@server ~]# iptables -A OUTPUT -p tcp --dport 443 -o eth0 -j LOG
[root@server ~]# iptables -A OUTPUT -p tcp --dport 80 -o eth0 -j REJECT
[root@server ~]# iptables -A OUTPUT -p tcp --dport 443 -o eth0 -j REJECT
[root@server ~]# iptables -L -n -v 
Chain INPUT (policy ACCEPT 123M packets, 81G bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 105M packets, 50G bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 LOG        tcp  --  *      eth0    0.0.0.0/0            0.0.0.0/0           tcp dpt:80 LOG flags 0 level 4 
   87  5205 LOG        tcp  --  *      eth0    0.0.0.0/0            0.0.0.0/0           tcp dpt:443 LOG flags 0 level 4 
    0     0 REJECT     tcp  --  *      eth0    0.0.0.0/0            0.0.0.0/0           tcp dpt:80 reject-with icmp-port-unreachable 
   17  2264 REJECT     tcp  --  *      eth0    0.0.0.0/0            0.0.0.0/0           tcp dpt:443 reject-with icmp-port-unreachable 


   e) get loaders

[root@server ~]# cobbler get-loaders
task started: 2010-12-30_180723_get_loaders
task started (id=Download Bootloader Content, time=Thu Dec 30 18:07:23 2010)
downloading http://mdehaan.fedorapeople.org/loaders/README to /var/lib/cobbler/loaders/README
downloading http://mdehaan.fedorapeople.org/loaders/COPYING.elilo to /var/lib/cobbler/loaders/COPYING.elilo
downloading http://mdehaan.fedorapeople.org/loaders/COPYING.yaboot to /var/lib/cobbler/loaders/COPYING.yaboot
downloading http://mdehaan.fedorapeople.org/loaders/COPYING.syslinux to /var/lib/cobbler/loaders/COPYING.syslinux
downloading http://mdehaan.fedorapeople.org/loaders/elilo-3.8-ia64.efi to /var/lib/cobbler/loaders/elilo-ia64.efi
downloading http://mdehaan.fedorapeople.org/loaders/yaboot-1.3.14-12 to /var/lib/cobbler/loaders/yaboot
downloading http://mdehaan.fedorapeople.org/loaders/pxelinux.0-3.61 to /var/lib/cobbler/loaders/pxelinux.0
downloading http://mdehaan.fedorapeople.org/loaders/menu.c32-3.61 to /var/lib/cobbler/loaders/menu.c32
*** TASK COMPLETE ***
 
   f) files downloaded
 
[root@server ~]# tree -upgt /var/lib/cobbler/loaders/
/var/lib/cobbler/loaders/
|-- [-rw-r--r-- root     root    ]  yaboot-1.3.14-12
|-- [-rw-r--r-- root     root    ]  pxelinux.0-3.61
|-- [-rw-r--r-- root     root    ]  menu.c32-3.61
|-- [-rw-r--r-- root     root    ]  COPYING.syslinux
|-- [-rw-r--r-- root     root    ]  README
|-- [-rw-r--r-- root     root    ]  elilo-3.8-ia64.efi
|-- [-rw-r--r-- root     root    ]  COPYING.yaboot
`-- [-rw-r--r-- root     root    ]  COPYING.elilo

0 directories, 8 files
[root@server ~]# 


Proxy Output: 
  $ tail -f /var/log/squid/access.log

1293760629.586     61 10.10.56.126 TCP_MISS/200 1532 GET http://mdehaan.fedorapeople.org/loaders/README - DIRECT/128.197.185.45 text/plain
1293760629.645     53 10.10.56.126 TCP_MISS/200 1115 GET http://mdehaan.fedorapeople.org/loaders/COPYING.elilo - DIRECT/128.197.185.45 text/plain
1293760629.710     53 10.10.56.126 TCP_MISS/200 1110 GET http://mdehaan.fedorapeople.org/loaders/COPYING.yaboot - DIRECT/128.197.185.45 text/plain
1293760629.829    113 10.10.56.126 TCP_MISS/200 18494 GET http://mdehaan.fedorapeople.org/loaders/COPYING.syslinux - DIRECT/128.197.185.45 text/plain
1293760630.133    302 10.10.56.126 TCP_MISS/200 356982 GET http://mdehaan.fedorapeople.org/loaders/elilo-3.8-ia64.efi - DIRECT/128.197.185.45 text/plain
1293760630.414    278 10.10.56.126 TCP_MISS/200 198681 GET http://mdehaan.fedorapeople.org/loaders/yaboot-1.3.14-12 - DIRECT/128.197.185.45 text/plain
1293760630.557    139 10.10.56.126 TCP_MISS/200 15203 GET http://mdehaan.fedorapeople.org/loaders/pxelinux.0-3.61 - DIRECT/128.197.185.45 text/plain
1293760630.753    192 10.10.56.126 TCP_MISS/200 39675 GET http://mdehaan.fedorapeople.org/loaders/menu.c32-3.61 - DIRECT/128.197.185.45 text/plain
Comment 3 Marcelo Moreira de Mello 2011-01-13 14:30:27 EST
Created attachment 473400 [details]
Patch for /etc/init.d/cobblerd

Hello, 

I made some changes on our spec file, which includes an option to read the /etc/sysconfig/cobblerd to set the http_proxy variable for read. This way guarantee that variables will be loaded when cobblerd  service will be load.  

$ cat /etc/sysconfig/cobblerd 
export HTTP_PROXY="http://proxy_server:3128"

$ service cobblerd restart

$ cobbler get-loaders 

Please, add this on the next cobblerd package for Red Hat. 

Best, 
Marcelo Moreira de Mello
Comment 4 Marcelo Moreira de Mello 2011-01-13 14:35:15 EST
Hello, 

Just to share that patch were committed in upstream. 

$> git show 0218888f9e72a6ffb358003f19686b040d7fd02a
commit 0218888f9e72a6ffb358003f19686b040d7fd02a
Author: Scott Henson <shenson@redhat.com>
Date:   Wed Jan 12 12:08:22 2011 -0500

    Allow dlcontent to use proxy environment variables

diff --git a/cobbler/action_dlcontent.py b/cobbler/action_dlcontent.py
index b3adae0..d73f114 100644
--- a/cobbler/action_dlcontent.py
+++ b/cobbler/action_dlcontent.py
@@ -62,13 +62,33 @@ class ContentDownloader:
           ( "%s/grub-0.97-x86_64.efi" % content_server, "%s/grub-x86_64.efi" % dest),
        )
 
-       self.logger.info("downloading content required to netboot all arches")
-       for f in files:
-          src = f[0]
-          dst = f[1]
-          if os.path.exists(dst) and not force:
-             self.logger.info("path %s already exists, not overwriting existing content, use --force if you wish to update" % dst)
-             continue
+       proxies = {} 
+       for var in os.environ.keys():
+          if (var.lower() == "http_proxy"):
+             proxies['http'] = os.environ[var]
+
+          if (var.lower() == "ftp_proxy"):
+             proxies['ftp'] = os.environ[var]
+
+       if (len(proxies) > 0):
+          for f in files:
+             src = f[0]
+             dst = f[1]
+             if os.path.exists(dst) and not force:
+                self.logger.info("path %s already exists, not overwriting existing content, use --force if you wish to update" % dst)
+                continue
+             os.chdir(dest)
+             self.logger.info("downloading %s to %s" % (src,dst))
+             urlgrabber.grabber.urlgrab(src, proxies=proxies)  
+
+       else:
+          self.logger.info("downloading content required to netboot all arches")
+          for f in files:
+             src = f[0]
+             dst = f[1]
+             if os.path.exists(dst) and not force:
+                self.logger.info("path %s already exists, not overwriting existing content, use --force if you wish to update" % dst)
+                continue
           self.logger.info("downloading %s to %s" % (src,dst))
           urlgrabber.urlgrab(src,dst)
Comment 9 Marcelo Moreira de Mello 2011-01-17 05:48:51 EST
Hello, 

Since the patch already were committed in upstream, for futures cobbler packages shipped by Red Hat honour the fix, we probably will just need to add in our SPEC the https://bugzilla.redhat.com/attachment.cgi?id=473400 and a new NOTE in our documentation that proxy is supported by doing a configuration in /etc/sysconfig/cobblerd. 

Best Regards, 
Marcelo Moreira de Mello
Comment 12 Clifford Perry 2011-01-19 12:32:22 EST
Lana - within changes made for bug 492961 - which was being used to track corrections to errors in the cobbler documentation, it seems we added a section for 'cobbler buildiso' usage by mistake (which was a proposed addition). 

This has caused this bug - bug 666435 & bug 667924 to be reported against Satellite 5.4.0. 

As noted within 506485 we have not spent time for development or QE in testing this functionality to flesh it out, as also shown by these two new bug reports. 

I am converting this into a documentation bug for Satellite 5.4.1 release cycle. Can we please use it to track the removal of cobbler buildiso from our reference guide. 

I'm made this comment public to allow GSS and others to easily find this. I've also changed the subject of the bug to reflect the change. 

Regards,
Clifford
Comment 13 Gilles Dubreuil 2011-01-31 07:30:53 EST
This could not be a documentation bug as this feature exists and is supported in Satellite 5.3.

As a matter of fact we have customers using the feature and not supporting will mean breaking their current build scenarios.

That feature works very well and comes straight for Spacewalk.

It seems that few things were moved around between two versions of Spacewalk which occured between Satellite 5.3 to 5.4.

The work around I found to fix it in Satellite 5.4 is:

Used /var/lib/cobbler/menu.c32 from RHN Satellite 5.3 and copied into RHN Satellite 5.4 as /var/lib/cobbler/loaders/menu.c32

After that Cobbler build iso works:

# cobbler buildiso
task started: 2010-11-02_182138_buildiso
task started (id=Build Iso, time=Tue Nov 2 18:21:38 2010)
using/creating tempdir: /buildiso
building tree for isolinux
copying miscellaneous files
copying kernels and initrds for profiles
generating a isolinux.cfg
generating profile list
done writing config
running: mkisofs -o /root/generated.iso -r -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -V Cobbler\ Install -R -J -T /buildiso
received:
ISO build complete
You may wish to delete: /buildiso
The output file is: /root/generated.iso
*** TASK COMPLETE *** 

Regards

Gilles
Comment 15 Aurelien Gouny 2011-02-01 22:06:58 EST
What happened to "we support what we ship" ?

I've just hit the exact same buildiso issue at a major customer & used Gilles' fix to be able to correct this after a Satellite upgrade, the customer has been using 'cobbler buildiso' heavily since 5.3 because of network restrictions due to security, it is my opinion that we can't just remove functions that we've shipped previously.

Thanks Gilles.
Aurelien (GPS Consultant - Sydney, AU)
Comment 16 Aurelien Gouny 2011-02-01 22:15:24 EST
To be exact and to detail further:
- compared to Gilles, I didn't have any /var/lib/cobbler/menu.c32
- tried 'cobbler get-loaders' but as the system is behind a proxy -> bug
- copied /usr/lib/syslinux/menu.c32 to /var/lib/cobbler/loaders/menu.c32 -> ok

'cobbler buildiso' now works fine.
Aurelien (GPS Consultant - Sydney, AU)
Comment 17 Gilles Dubreuil 2011-02-01 23:27:28 EST
It's more than just a doco mistake. Because of the impact on customers

The functionality was in 5.3 and documented and still is in 5.4 and documented.
Now if one cannot build servers with PXE Boot than we have to provide a way to build ISOs.

Regards,

Gilles
Comment 18 Lana Brindley 2011-03-21 23:26:37 EDT
In the Kickstart chapter in the new User Guide, I have included the following procedure:

<procedure id="proc-Getting_Started_Guide-Kickstarting_from_Bare_Metal_-Cobbler_Boot_ISO">
<title>Cobbler Boot ISO</title>
	 <para>
		The Cobbler boot ISO is a disk image that can be built on a &SAT; server and burned to a CD or DVD. It can then be used to boot a bare metal system.
	</para>
<step>
	<para>
		To create a boot ISO, log in to the &SAT; server as root and run <command>cobbler buildiso</command>. The ISO will contain all kernel and initrd images stored in the &SAT;, along with all associated kernel argument settings.
	</para>
	 <para>
		Kickstart files will be sourced remotely, which means that changes to the kickstart templates can be made without having to re-burn the disk. However, if you create a new kickstart profile and want to use it with the Cobbler boot ISO, you will need to create a new disk.
	</para>
</step>
<step>
	<para>
		Insert the Cobbler boot ISO disk. A menu of available kickstarts will be displayed. Select the appropriate kickstart and the system will start to install automatically. <remark>A screenshot here would be good. LKB</remark>
	</para>
	 <note>
	<title>Note</title>
	 <para>
		Any time you add a kickstart within Satellite, you will need to recreate the ISO and re-burn it to an optical disk.
	</para>
	</note>
</step>
</procedure>

For the purposes of fixing this bug, I have commented the procedure out. If it is decided later on to have this procedure visible, please raise a new bug.

LKB
Comment 21 Pavel Novotny 2011-04-21 08:24:53 EDT
Taking this bug for verification.
Comment 25 Lana Brindley 2011-05-05 20:18:56 EDT
This book has now been dropped to translation (RT#75265).
No further updates can be accepted. Please raise a new bug for any changes.
LKB
Comment 26 Lana Brindley 2011-06-16 18:14:06 EDT
5.4.1 Satellite books are now available on docs.redhat.com. Please raise a new bug for any issues.

LKB
Comment 27 Lana Brindley 2011-06-21 23:51:39 EDT
Reopening, as Cliff found some more instances.
Comment 28 Lana Brindley 2011-06-22 00:28:53 EDT
Found and fixed several instances in 5.3 and 5.4 Deployment Guides.

LKB
Comment 29 Lana Brindley 2011-06-22 00:32:13 EDT
Requested respin in RT#114048

LKB

Note You need to log in before you can comment on or make changes to this bug.