Bug 865841

Summary: When several clients request updates some clients can't get packages
Product: [Retired] Subscription Asset Manager Reporter: Og Maciel <omaciel>
Component: thumbslugAssignee: Katello Bug Bin <katello-bugs>
Status: CLOSED ERRATA QA Contact: Tazim Kolhar <tkolhar>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 1.3CC: bkearney, cwelton, sthirugn, tkolhar
Target Milestone: rc   
Target Release: 1.3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-10-01 10:50:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 833466    

Description Og Maciel 2012-10-12 15:37:11 UTC
Description of problem:

So I have a SAM 1.2 system with 8 real clients properly registered and subscribed, running alongside several other VMS... it can get a bit slow for sure. Anyhow, I tried to install httpd in all of them inside a loop:

for i in 2 3 4 5 6 7 8 9; do ssh root@qeclient0$i.usersys.redhat.com 'yum install -y httpd'; done

Some of the clients got httpd installed, some failed with:

Downloading Packages:
https://qetello03.usersys.redhat.com:8088/content/dist/rhel/server/6/6Server/x86_64/os/Packages/apr-1.3.9-5.el6_2.x86_64.rpm: [Errno 14] problem making ssl connection
Trying other mirror.


Error Downloading Packages:
  apr-1.3.9-5.el6_2.x86_64: failure: Packages/apr-1.3.9-5.el6_2.x86_64.rpm from rhel-6-server-rpms: [Errno 256] No more mirrors to try.

Version-Release number of selected component (if applicable):

* candlepin-0.7.13-1.el6_3.noarch
* candlepin-tomcat6-0.7.13-1.el6_3.noarch
* katello-candlepin-cert-key-pair-1.0-1.noarch
* katello-certs-tools-1.1.8-1h.el6_3.noarch
* katello-cli-1.1.11-1h.el6_3.noarch
* katello-cli-common-1.1.11-1h.el6_3.noarch
* katello-common-1.1.15-2h.el6_3.noarch
* katello-configure-1.1.11-1h.el6_3.noarch
* katello-glue-candlepin-1.1.15-2h.el6_3.noarch
* katello-headpin-1.1.15-2h.el6_3.noarch
* katello-headpin-all-1.1.15-2h.el6_3.noarch
* katello-selinux-1.1.2-1h.el6_3.noarch
* thumbslug-0.0.24-1.el6_2.noarch
* thumbslug-selinux-0.0.24-1.el6_2.noarch

How reproducible:


Steps to Reproduce:
1. Registered and subscribed 8 clients (all RHEL 62. x86_64) to my organization
2. Upload valid manifest
3. get all 8 clients to install something, such as firefox or httpd
  
Actual results:


Expected results:


Additional info:

fwiw this is a SAM 1.2 VM with 4GB RAM and sharing space with other VMS.

I also noticed thumbslug:

Oct 11 16:21:47 [New I/O server worker #1-6] ERROR org.candlepin.thumbslug.HttpRequestHandler - Exception caught!
java.io.IOException: Connection reset by peer
»···at sun.nio.ch.FileDispatcher.read0(Native Method)
»···at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
»···at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:251)
»···at sun.nio.ch.IOUtil.read(IOUtil.java:218)
»···at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:254)
»···at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
»···at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
»···at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
»···at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
»···at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
»···at java.lang.Thread.run(Thread.java:679)
Oct 11 17:00:58 [New I/O server worker #1-2] ERROR org.candlepin.thumbslug.HttpRequestHandler - Exception caught!
java.io.IOException: Broken pipe
»···at sun.nio.ch.FileDispatcher.write0(Native Method)
»···at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
»···at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122)
»···at sun.nio.ch.IOUtil.write(IOUtil.java:78)
»···at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:352)
»···at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$PooledSendBuffer.transferTo(SocketSendBufferPool.java:240)
»···at org.jboss.netty.channel.socket.nio.NioWorker.write0(NioWorker.java:472)
»···at org.jboss.netty.channel.socket.nio.NioWorker.writeFromSelectorLoop(NioWorker.java:402)
»···at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:288)
»···at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
»···at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
»···at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
»···at java.lang.Thread.run(Thread.java:679)

catalina.out shows:

Oct 12 10:52:15 [http-8443-8] WARN  org.candlepin.jackson.JsonBeanPropertyFilter - Unable to serialize property 'autoheal without getter: getAutoheal
Oct 12 10:52:16 [http-8443-9] WARN  org.candlepin.jackson.JsonBeanPropertyFilter - Unable to serialize property 'canActivate without getter: getCanActivate
Oct 12 10:52:16 [http-8443-9] WARN  org.candlepin.jackson.JsonBeanPropertyFilter - Unable to serialize property 'autoheal without getter: getAutoheal
Oct 12 10:52:16 [http-8443-6] WARN  org.candlepin.jackson.JsonBeanPropertyFilter - Unable to serialize property 'canActivate without getter: getCanActivate
Oct 12 10:52:16 [http-8443-6] WARN  org.candlepin.jackson.JsonBeanPropertyFilter - Unable to serialize property 'autoheal without getter: getAutoheal
Oct 12 10:52:16 [http-8443-2] WARN  org.candlepin.jackson.JsonBeanPropertyFilter - Unable to serialize property 'canActivate without getter: getCanActivate
Oct 12 10:52:16 [http-8443-2] WARN  org.candlepin.jackson.JsonBeanPropertyFilter - Unable to serialize property 'autoheal without getter: getAutoheal

Comment 1 Og Maciel 2012-10-25 14:04:11 UTC
I tried this same experiment with a "beefy" VM that had 3 times as much RAM and using latest CFSE. After registering and subscribing 10 systems, I had no issue consuming content when making 10 simultaneous requests. Will have to perform the same experiment using SAM with more RAM (or less VMs competing for resources).

Comment 2 Og Maciel 2012-10-31 18:28:14 UTC
Re-tested SAM 1.2 with 10 simultaneous requests, all passed w/o a glitch

Comment 3 James Bowes 2012-11-07 17:24:12 UTC
I've done some more work to help with concurrency in thumbslug, though ultimately we're limited by your system's cpu's ability do process ssl connections, and how many concurrent connections from the same system the CDN will accept.

Anyhoo, calling this one FIXED in thumbslug master, 01f85744de1ef3e12373b065
should be in thumbslug 0.0.27

Comment 4 Bryan Kearney 2013-06-07 18:20:17 UTC
Moving all POST bugs to ON_QA since we have delivered a puddle with the bugs.

Comment 5 Tazim Kolhar 2013-08-13 16:20:49 UTC
VERIFIED :

# rpm -qa | grep katello
katello-selinux-1.4.4-2.el6sat.noarch
katello-candlepin-cert-key-pair-1.0-1.noarch
katello-certs-tools-1.4.2-2.el6sat.noarch
katello-cli-common-1.4.3-5.el6sat.noarch
katello-cli-1.4.3-5.el6sat.noarch
katello-common-1.4.3-6.el6sam_splice.noarch
katello-configure-1.4.4-2.el6sat.noarch
katello-glue-elasticsearch-1.4.3-6.el6sam_splice.noarch
katello-headpin-all-1.4.3-6.el6sam_splice.noarch
katello-glue-candlepin-1.4.3-6.el6sam_splice.noarch
signo-katello-0.0.10-2.el6sat.noarch
katello-headpin-1.4.3-6.el6sam_splice.noarch


1.)Upload valid manifest on Sam server

2.)Registered and subscribed 8 clients (all RHEL 62. x86_64) to my organization
   
    i. rpm -Uvh http://SAM_SERVER/pub/candlepin-cert-consumer-latest.noarch.rpm
Retrieving http://SAM_SERVER/pub/candlepin-cert-consumer-latest.noarch.rpm
Preparing...                ########################################### [100%]
   1:candlepin-cert-consumer########################################### [100%]


   ii.# subscription-manager register --org=ACME_Corporation --autosubscribe


3.) got all 8 clients to install firefox
    
   
4.) run firefox &  
    # firefox &
      [1] 15297

Comment 7 errata-xmlrpc 2013-10-01 10:50:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2013-1390.html