1414697 – cdn-sync makes direct connections even with --http-proxy option specified.

Bug 1414697 - cdn-sync makes direct connections even with --http-proxy option specified.

Summary: cdn-sync makes direct connections even with --http-proxy option specified.

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat Satellite 5
Classification:	Red Hat
Component:	Satellite Synchronization
Sub Component:
Version:	580
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Assignee:	Jan Dobes
QA Contact:	Patrik Segedy
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1400201
TreeView+	depends on / blocked

Reported:	2017-01-19 09:52 UTC by Radovan Drazny
Modified:	2017-06-21 12:08 UTC (History)
CC List:	3 users (show)
Fixed In Version:	spacewalk-backend-2.5.3-52
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-06-21 12:08:52 UTC
Target Upstream Version:

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Description Radovan Drazny 2017-01-19 09:52:27 UTC

Description of problem:
When using cdn-sync with --http-proxy option specified, connections to CDN are still direct, and not via the proxy. 

Version-Release number of selected component (if applicable):
spacewalk-backend-cdn-2.5.3-49

How reproducible:
always

Steps to Reproduce:
Two different ways to observe connections
First way:

1. Block all outgoing traffic from the server to https.

   iptables -A OUTPUT -p tcp --dport 443 -j REJECT

2. Run the cdn-sync with --http-proxy specified.

   #cdn-sync -vv -c rhel-x86_64-server-7 --http-proxy squid.redhat.com:3128
   <...>
   04:15:07 7973/13807 : lvm2-python-libs-2.02.130-5.el7_2.1.x86_64.rpm (failed)
   04:15:07 7974/13807 : lzo-2.06-6.el7.x86_64.rpm (failed)
   04:15:07 ERROR: Download failed: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/Packages/lzo-2.06-8.el7.i686.rpm - [Errno 14] PYCURL ERROR 7 - "couldn't connect to host".
   04:15:07 7975/13807 : lzo-2.06-8.el7.i686.rpm (failed)
   04:15:07 ERROR: Download failed: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/Packages/lzo-2.06-8.el7.x86_64.rpm - [Errno 14] PYCURL ERROR 7 - "couldn't connect to host".
   04:15:07 ERROR: Download failed: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/Packages/lzo-2.06-6.el7_0.2.x86_64.rpm - [Errno 14] PYCURL ERROR 7 - "couldn't connect to host".
   04:15:07 7976/13807 : lzo-2.06-6.el7_0.2.x86_64.rpm (failed)
   04:15:07 7977/13807 : lzo-2.06-8.el7.x86_64.rpm (failed)
   04:15:08 ERROR: Download failed: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/Packages/lzo-2.06-6.el7_0.2.i686.rpm - [Errno 14] PYCURL ERROR 7 - "couldn't connect to host".
   04:15:08 ERROR: Download failed: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/Packages/lzo-2.06-6.el7.i686.rpm - [Errno 14] PYCURL ERROR 7 - "couldn't connect to host".
   04:15:08 7978/13807 : lzo-2.06-6.el7.i686.rpm (failed)
   04:15:08 7979/13807 : lzo-2.06-6.el7_0.2.i686.rpm (failed)
   <...>

3. Connections should be made using the proxy, which is not blocked. cdn-sync attempts to connect directly instead, and fails, because of the firewall rule.

The second way (no firewall rule needed).

1. Run following command in one terminal on the server:

   #watch -n 1 'lsof -i | grep cdn-sync'

2. In an other terminal run cdn-sync with --http-proxy option

   #cdn-sync -vv -c rhel-x86_64-server-7 --http-proxy squid.redhat.com:3128

3. Watch output of the command from the step 1. First connection to get the metadata is made using the proxy:

   cdn-sync  29078     root   11r  IPv4 598049      0t0  TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:35798->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED)
   
   But data connections for the packages itself are made direct, without using the proxy:
   
   <...>
   Every 1.0s: lsof -i | grep cdn-sync                                                                                                        Thu Jan 19 04:26:24 2017

   cdn-sync  29078     root   15r  IPv4 623022      0t0  TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:46398->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED)
   cdn-sync  29078     root   17u  IPv4 623027      0t0  TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:46400->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED)
   cdn-sync  29078     root   18u  IPv4 623020      0t0  TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:46396->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED)
   cdn-sync  29078     root   19u  IPv4 623029      0t0  TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:46402->a95-101-44-251.deploy.akamaitechnologies.com:https (SYN_SENT)
   cdn-sync  29078     root   20u  IPv4 623018      0t0  TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:46394->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED)
   <...>
   Every 1.0s: lsof -i | grep cdn-sync                                                                                                       Thu Jan 19 04:26:39 2017
 
   cdn-sync  29078     root   15r  IPv4 623340      0t0  TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:46466->a95-101-44-251.deploy.akamaitechnologies.com:https (SYN_SENT)
   cdn-sync  29078     root   17u  IPv4 623334      0t0  TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:46462->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED)
   cdn-sync  29078     root   18u  IPv4 623287      0t0  TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:46458->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED)
   cdn-sync  29078     root   19u  IPv4 623337      0t0  TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:46464->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED)
   cdn-sync  29078     root   20u  IPv4 623289      0t0  TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:46460->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED)
   <...>
   Every 1.0s: lsof -i | grep cdn-sync                                                                                                       Thu Jan 19 04:29:38 2017

   cdn-sync  29078     root   14r  IPv4 626064      0t0  TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:47818->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED)
   cdn-sync  29078     root   16u  IPv4 626066      0t0  TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:47820->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED)
   cdn-sync  29078     root   17u  IPv4 626070      0t0  TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:44382->a173-222-212-251.deploy.static.akamaitechnologies.com:https (ESTABLISHED)
   cdn-sync  29078     root   18u  IPv4 625510      0t0  TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:47812->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED)
   cdn-sync  29078     root   20u  IPv4 626058      0t0  TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:47814->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED)

Connections are made directly to the CDN, no proxy in sight. 
If you combine both methods - the firewall rule and connections watching - you can observe unsuccessful connections to the CDN:

   Every 1.0s: lsof -i | grep cdn-sync                                                                                                       Thu Jan 19 04:40:03 2017

   cdn-sync  30125     root   16u  IPv4 661087      0t0  TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:48164->a173-222-212-251.deploy.static.akamaitechnologies.com:https (SYN_SENT)
   cdn-sync  30125     root   17u  IPv4 661089      0t0  TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:48166->a173-222-212-251.deploy.static.akamaitechnologies.com:https (SYN_SENT)
   cdn-sync  30125     root   18u  IPv4 661094      0t0  TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:48170->a173-222-212-251.deploy.static.akamaitechnologies.com:https (SYN_SENT)
   cdn-sync  30125     root   19u  IPv4 661091      0t0  TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:48168->a173-222-212-251.deploy.static.akamaitechnologies.com:https (SYN_SENT)
   cdn-sync  30125     root   20u  IPv4 661095      0t0  TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:48172->a173-222-212-251.deploy.static.akamaitechnologies.com:https (SYN_SENT)

Connections never progress past the SYN_SENT phase, and cdn-sync output is full of errors about "couldn't connect to host".

Actual results:
Connections are made directly, without proxy involved.

Expected results:

No direct connection performed, all communication with the CDN should be done via specified proxy.

Additional info:

Comment 1 Pavel Studeník 2017-01-19 10:22:05 UTC

I have problem to use parameter http-proxy. Look at my
bug 1412624 - cdn-sync with parameter http-proxy causes ERROR: invalid arguments to setopt

Comment 2 Radovan Drazny 2017-01-19 10:45:13 UTC

(In reply to Pavel Studeník from comment #1)
> I have problem to use parameter http-proxy. Look at my
> bug 1412624 - cdn-sync with parameter http-proxy causes ERROR: invalid
> arguments to setopt

Yes, I'm aware of bug 1412624. I have fixed it manually on my box using the solution described in the comment #2.

Comment 3 Jan Dobes 2017-01-19 16:13:25 UTC

fixed in spacewalk master:

1d9cf820d81b87727156a174c0a05c7575e6c674

Comment 6 Jan Dobes 2017-01-23 12:30:02 UTC

fixing pylint, spacewalk master:

f0db2e167f106347f4c06dd6492d38a628c29c22

Comment 9 Patrik Segedy 2017-01-25 14:09:08 UTC

Steps to verify:
1. Apply patch from bug 1412624, comment 1
2. In one terminal on server run
   #watch -n 1 'lsof -i | grep cdn-sync'

3. In the other terminal run cdn-sync with --http-proxy option
   #cdn-sync -vv -c rhel-x86_64-server-7 --http-proxy squid.redhat.com:3128

4. In output from step 2 that connections are made through proxy

cdn-sync   14808     root   11u  IPv4 587961	  0t0  TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:60966->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED)

cdn-sync   14808     root   14u  IPv4 592210	  0t0  TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:32790->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED)
cdn-sync   14808     root   17u  IPv4 592342	  0t0  TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:32802->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED)
cdn-sync   14808     root   18u  IPv4 592385	  0t0  TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:32804->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED)
cdn-sync   14808     root   19w  IPv4 592336	  0t0  TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:32798->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED)
cdn-sync   14808     root   20u  IPv4 592339	  0t0  TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:32800->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED)

5. When I block https as mentioned in reproducer, there are no errors in cdn-sync output and connections are made through proxy

iptables -A OUTPUT -p tcp --dport 443 -j REJECT

cdn-sync   16361     root   16u  IPv4 618033	  0t0  TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:34650->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED)
cdn-sync   16361     root   17w  IPv4 618005	  0t0  TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:34648->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED)
cdn-sync   16361     root   18w  IPv4 618040	  0t0  TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:34654->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED)
cdn-sync   16361     root   19u  IPv4 618042	  0t0  TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:34656->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED)
cdn-sync   16361     root   20u  IPv4 618037	  0t0  TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:34652->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED)

VERIFIED

Note You need to log in before you can comment on or make changes to this bug.