Description of problem: When using cdn-sync with --http-proxy option specified, connections to CDN are still direct, and not via the proxy. Version-Release number of selected component (if applicable): spacewalk-backend-cdn-2.5.3-49 How reproducible: always Steps to Reproduce: Two different ways to observe connections First way: 1. Block all outgoing traffic from the server to https. iptables -A OUTPUT -p tcp --dport 443 -j REJECT 2. Run the cdn-sync with --http-proxy specified. #cdn-sync -vv -c rhel-x86_64-server-7 --http-proxy squid.redhat.com:3128 <...> 04:15:07 7973/13807 : lvm2-python-libs-2.02.130-5.el7_2.1.x86_64.rpm (failed) 04:15:07 7974/13807 : lzo-2.06-6.el7.x86_64.rpm (failed) 04:15:07 ERROR: Download failed: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/Packages/lzo-2.06-8.el7.i686.rpm - [Errno 14] PYCURL ERROR 7 - "couldn't connect to host". 04:15:07 7975/13807 : lzo-2.06-8.el7.i686.rpm (failed) 04:15:07 ERROR: Download failed: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/Packages/lzo-2.06-8.el7.x86_64.rpm - [Errno 14] PYCURL ERROR 7 - "couldn't connect to host". 04:15:07 ERROR: Download failed: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/Packages/lzo-2.06-6.el7_0.2.x86_64.rpm - [Errno 14] PYCURL ERROR 7 - "couldn't connect to host". 04:15:07 7976/13807 : lzo-2.06-6.el7_0.2.x86_64.rpm (failed) 04:15:07 7977/13807 : lzo-2.06-8.el7.x86_64.rpm (failed) 04:15:08 ERROR: Download failed: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/Packages/lzo-2.06-6.el7_0.2.i686.rpm - [Errno 14] PYCURL ERROR 7 - "couldn't connect to host". 04:15:08 ERROR: Download failed: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/Packages/lzo-2.06-6.el7.i686.rpm - [Errno 14] PYCURL ERROR 7 - "couldn't connect to host". 04:15:08 7978/13807 : lzo-2.06-6.el7.i686.rpm (failed) 04:15:08 7979/13807 : lzo-2.06-6.el7_0.2.i686.rpm (failed) <...> 3. Connections should be made using the proxy, which is not blocked. cdn-sync attempts to connect directly instead, and fails, because of the firewall rule. The second way (no firewall rule needed). 1. Run following command in one terminal on the server: #watch -n 1 'lsof -i | grep cdn-sync' 2. In an other terminal run cdn-sync with --http-proxy option #cdn-sync -vv -c rhel-x86_64-server-7 --http-proxy squid.redhat.com:3128 3. Watch output of the command from the step 1. First connection to get the metadata is made using the proxy: cdn-sync 29078 root 11r IPv4 598049 0t0 TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:35798->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED) But data connections for the packages itself are made direct, without using the proxy: <...> Every 1.0s: lsof -i | grep cdn-sync Thu Jan 19 04:26:24 2017 cdn-sync 29078 root 15r IPv4 623022 0t0 TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:46398->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED) cdn-sync 29078 root 17u IPv4 623027 0t0 TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:46400->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED) cdn-sync 29078 root 18u IPv4 623020 0t0 TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:46396->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED) cdn-sync 29078 root 19u IPv4 623029 0t0 TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:46402->a95-101-44-251.deploy.akamaitechnologies.com:https (SYN_SENT) cdn-sync 29078 root 20u IPv4 623018 0t0 TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:46394->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED) <...> Every 1.0s: lsof -i | grep cdn-sync Thu Jan 19 04:26:39 2017 cdn-sync 29078 root 15r IPv4 623340 0t0 TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:46466->a95-101-44-251.deploy.akamaitechnologies.com:https (SYN_SENT) cdn-sync 29078 root 17u IPv4 623334 0t0 TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:46462->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED) cdn-sync 29078 root 18u IPv4 623287 0t0 TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:46458->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED) cdn-sync 29078 root 19u IPv4 623337 0t0 TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:46464->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED) cdn-sync 29078 root 20u IPv4 623289 0t0 TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:46460->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED) <...> Every 1.0s: lsof -i | grep cdn-sync Thu Jan 19 04:29:38 2017 cdn-sync 29078 root 14r IPv4 626064 0t0 TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:47818->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED) cdn-sync 29078 root 16u IPv4 626066 0t0 TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:47820->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED) cdn-sync 29078 root 17u IPv4 626070 0t0 TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:44382->a173-222-212-251.deploy.static.akamaitechnologies.com:https (ESTABLISHED) cdn-sync 29078 root 18u IPv4 625510 0t0 TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:47812->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED) cdn-sync 29078 root 20u IPv4 626058 0t0 TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:47814->a95-101-44-251.deploy.akamaitechnologies.com:https (ESTABLISHED) Connections are made directly to the CDN, no proxy in sight. If you combine both methods - the firewall rule and connections watching - you can observe unsuccessful connections to the CDN: Every 1.0s: lsof -i | grep cdn-sync Thu Jan 19 04:40:03 2017 cdn-sync 30125 root 16u IPv4 661087 0t0 TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:48164->a173-222-212-251.deploy.static.akamaitechnologies.com:https (SYN_SENT) cdn-sync 30125 root 17u IPv4 661089 0t0 TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:48166->a173-222-212-251.deploy.static.akamaitechnologies.com:https (SYN_SENT) cdn-sync 30125 root 18u IPv4 661094 0t0 TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:48170->a173-222-212-251.deploy.static.akamaitechnologies.com:https (SYN_SENT) cdn-sync 30125 root 19u IPv4 661091 0t0 TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:48168->a173-222-212-251.deploy.static.akamaitechnologies.com:https (SYN_SENT) cdn-sync 30125 root 20u IPv4 661095 0t0 TCP hp-dl180g6-01.rhts.eng.bos.redhat.com:48172->a173-222-212-251.deploy.static.akamaitechnologies.com:https (SYN_SENT) Connections never progress past the SYN_SENT phase, and cdn-sync output is full of errors about "couldn't connect to host". Actual results: Connections are made directly, without proxy involved. Expected results: No direct connection performed, all communication with the CDN should be done via specified proxy. Additional info:
I have problem to use parameter http-proxy. Look at my bug 1412624 - cdn-sync with parameter http-proxy causes ERROR: invalid arguments to setopt
(In reply to Pavel Studeník from comment #1) > I have problem to use parameter http-proxy. Look at my > bug 1412624 - cdn-sync with parameter http-proxy causes ERROR: invalid > arguments to setopt Yes, I'm aware of bug 1412624. I have fixed it manually on my box using the solution described in the comment #2.
fixed in spacewalk master: 1d9cf820d81b87727156a174c0a05c7575e6c674
fixing pylint, spacewalk master: f0db2e167f106347f4c06dd6492d38a628c29c22
Steps to verify: 1. Apply patch from bug 1412624, comment 1 2. In one terminal on server run #watch -n 1 'lsof -i | grep cdn-sync' 3. In the other terminal run cdn-sync with --http-proxy option #cdn-sync -vv -c rhel-x86_64-server-7 --http-proxy squid.redhat.com:3128 4. In output from step 2 that connections are made through proxy cdn-sync 14808 root 11u IPv4 587961 0t0 TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:60966->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED) cdn-sync 14808 root 14u IPv4 592210 0t0 TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:32790->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED) cdn-sync 14808 root 17u IPv4 592342 0t0 TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:32802->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED) cdn-sync 14808 root 18u IPv4 592385 0t0 TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:32804->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED) cdn-sync 14808 root 19w IPv4 592336 0t0 TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:32798->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED) cdn-sync 14808 root 20u IPv4 592339 0t0 TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:32800->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED) 5. When I block https as mentioned in reproducer, there are no errors in cdn-sync output and connections are made through proxy iptables -A OUTPUT -p tcp --dport 443 -j REJECT cdn-sync 16361 root 16u IPv4 618033 0t0 TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:34650->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED) cdn-sync 16361 root 17w IPv4 618005 0t0 TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:34648->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED) cdn-sync 16361 root 18w IPv4 618040 0t0 TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:34654->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED) cdn-sync 16361 root 19u IPv4 618042 0t0 TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:34656->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED) cdn-sync 16361 root 20u IPv4 618037 0t0 TCP dell-per630-fc-01.rhts.eng.bos.redhat.com:34652->proxy01.intranet.prod.int.rdu2.redhat.com:squid (ESTABLISHED) VERIFIED