Bug 973181 - RestClient::RequestTimeout on publishing content view
Summary: RestClient::RequestTimeout on publishing content view
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Content Management
Version: 6.0.1
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: Unspecified
Assignee: Katello Bug Bin
QA Contact: Og Maciel
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-06-11 12:34 UTC by Garik Khachikyan
Modified: 2015-01-04 21:59 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-06-13 13:28:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
logs (2.72 MB, application/gzip)
2013-06-11 12:34 UTC, Garik Khachikyan
no flags Details
UI exception (47.13 KB, image/png)
2013-06-11 12:41 UTC, Garik Khachikyan
no flags Details

Description Garik Khachikyan 2013-06-11 12:34:55 UTC
Created attachment 759605 [details]
logs

Description of problem:
My tries publishing content view that contains synced RHEL 6.3 and RHEL 6.4 repos (64bit) always are ending with the "RestClient::RequestTimeout" exception and failing.

Version-Release number of selected component (if applicable):
candlepin-0.8.9-1.el6_4.noarch
candlepin-cert-consumer-sun-x4440-01.rhts.eng.bos.redhat.com-1.0-1.noarch
candlepin-scl-1-5.el6_4.noarch
candlepin-scl-quartz-2.1.5-5.el6_4.noarch
candlepin-scl-rhino-1.7R3-1.el6_4.noarch
candlepin-scl-runtime-1-5.el6_4.noarch
candlepin-selinux-0.8.9-1.el6_4.noarch
candlepin-tomcat6-0.8.9-1.el6_4.noarch
elasticsearch-0.19.9-8.el6sat.noarch
katello-1.4.2-11.el6sat.noarch
katello-agent-1.4.3-1.git.1.24fe511.el6.noarch
katello-all-1.4.2-11.el6sat.noarch
katello-candlepin-cert-key-pair-1.0-1.noarch
katello-certs-tools-1.4.2-2.el6sat.noarch
katello-cli-1.4.2-7.el6sat.noarch
katello-cli-common-1.4.2-7.el6sat.noarch
katello-common-1.4.2-11.el6sat.noarch
katello-configure-1.4.3-14.el6sat.noarch
katello-configure-foreman-1.4.3-14.el6sat.noarch
katello-foreman-all-1.4.2-11.el6sat.noarch
katello-glue-candlepin-1.4.2-11.el6sat.noarch
katello-glue-elasticsearch-1.4.2-11.el6sat.noarch
katello-glue-pulp-1.4.2-11.el6sat.noarch
katello-qpid-broker-key-pair-1.0-1.noarch
katello-qpid-client-key-pair-1.0-1.noarch
katello-selinux-1.4.3-3.el6sat.noarch
m2crypto-0.21.1.pulp-8.el6sat.x86_64
mod_wsgi-3.4-1.pulp.el6sat.x86_64
pulp-rpm-handlers-2.1.1-1.el6sat.noarch
pulp-rpm-plugins-2.1.1-1.el6sat.noarch
pulp-selinux-2.1.1-1.el6sat.noarch
pulp-server-2.1.1-1.el6sat.noarch
python-isodate-0.5.0-1.pulp.el6sat.noarch
python-oauth2-1.5.170-3.pulp.el6sat.noarch
python-pulp-agent-lib-2.1.1-1.el6sat.noarch
python-pulp-common-2.1.1-1.el6sat.noarch
python-pulp-rpm-common-2.1.1-1.el6sat.noarch
python-qpid-0.18-5.el6_4.noarch
python-rhsm-1.8.0-1.pulp.el6sat.x86_64
qpid-cpp-client-0.14-22.el6_3.x86_64
qpid-cpp-client-ssl-0.14-22.el6_3.x86_64
qpid-cpp-server-0.14-22.el6_3.x86_64
qpid-cpp-server-ssl-0.14-22.el6_3.x86_64
ruby193-rubygem-foreman-katello-engine-0.0.8-5.el6sat.noarch
ruby193-rubygem-katello-foreman-engine-0.0.3-4.el6sat.noarch
ruby193-rubygem-katello_api-0.0.3-2.el6_4.noarch
ruby193-rubygem-ldap_fluff-0.2.2-1.el6sat.noarch
signo-katello-0.0.16-1.el6sat.noarch

How reproducible:
3 from 3 attempts.

Steps to Reproduce (I tried all through UI):
1.import valid manifest
2.enable RHEL6 x86_64 repos (6.1 - 6.4, 6Server)
3.sync RHEL 6.3 and RHEL 6.4 repos
4. make content view definition and add that 2 synced repos
5. try to publish it

Actual results:
after running about 10 min it fails (see above)

Expected results:
no failure. Content should be published making it possible to promote to next environments and consume it!

Additional info:
logs are attached

Please not that this could have critical impact on our clients blocking them from consuming Red Hat content.

Comment 1 Garik Khachikyan 2013-06-11 12:41:18 UTC
Created attachment 759606 [details]
UI exception

Comment 2 Og Maciel 2013-06-11 13:17:03 UTC
How much RAM does your server have?

Comment 3 Garik Khachikyan 2013-06-11 13:20:41 UTC
12GB and configured via:
---
-katello-web-workers=2 --job-workers=2 --es-min-mem=512M --es-max-mem=1024M

Comment 4 Corey Welton 2013-06-11 14:17:00 UTC
Seeing this as well in the latest puddle.

[root@mgmt9 ~]# cat /proc/meminfo
MemTotal:        8059328 kB
MemFree:          439728 kB
Buffers:          314148 kB
Cached:          2782796 kB
SwapCached:        44752 kB
Active:          4727948 kB
Inactive:        2411756 kB
Active(anon):    2932732 kB
Inactive(anon):  1119784 kB
Active(file):    1795216 kB
Inactive(file):  1291972 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       8191992 kB
SwapFree:        8070820 kB
Dirty:                68 kB
Writeback:             0 kB
AnonPages:       4019584 kB
Mapped:           324644 kB
Shmem:              9752 kB
Slab:             371520 kB
SReclaimable:     297356 kB
SUnreclaim:        74164 kB
KernelStack:        3328 kB
PageTables:        31816 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    12221656 kB
Committed_AS:    3762992 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       28180 kB
VmallocChunk:   34359700412 kB
HardwareCorrupted:     0 kB
AnonHugePages:   1611776 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        9600 kB
DirectMap2M:     8378368 kB

Comment 5 Corey Welton 2013-06-11 15:28:07 UTC
FWIW, after a third attempt to publish my CVD, I get the following error in the UI

Pulp has already been taken


katello production log:

[ERROR 2013-06-11 10:12:33 pulp_rest  #22431] RestClient.post "https://mgmt9.rhq.lab.eng.bos.redhat.com/pulp/api/v2/content/units/rpm/search/", 8142 byte(s) 
length, "Accept"=>"application/json", "Accept-Encoding"=>"gzip, deflate", "Authorization"=>"OAuth oauth_body_hash=\"2jmj7l5rSw0yVb%2FvlWAYkK%2FYBwk%3D\", oau
th_consumer_key=\"katello\", oauth_nonce=\"7tf5jJZrVdsYk0v7GcgD4aazr0Pn9cl3yC6PpzBS3g\", oauth_signature=\"2LYObsRBvQ1UeATHEmxFbeg6eCs%3D\", oauth_signature_
method=\"HMAC-SHA1\", oauth_timestamp=\"1370959893\", oauth_version=\"1.0\"", "Content-Length"=>"8142", "Content-Type"=>"application/json", "pulp-user"=>"adm
in"
"production.log" 506L, 53313C
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:78:in `call'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:78:in `block in add'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `call'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `execute'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:38:in `run_callbacks'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:127:in `start'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:101:in `run'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:89:in `block in run_process'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/application.rb:249:in `call'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/application.rb:249:in `block in start_proc'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/daemonize.rb:197:in `call'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/daemonize.rb:197:in `call_as_daemon'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/application.rb:253:in `start_proc'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/application.rb:293:in `start'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/controller.rb:70:in `run'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons.rb:195:in `block in run_proc'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/cmdline.rb:109:in `call'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/cmdline.rb:109:in `catch_exceptions'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons.rb:194:in `run_proc'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:87:in `run_process'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:80:in `block in daemonize'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:78:in `times'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:78:in `daemonize'
 | script/delayed_job:5:in `<main>'

Comment 6 Og Maciel 2013-06-11 15:46:50 UTC
https://bugzilla.redhat.com/show_bug.cgi?id=972852 is to track "Pulp has been taken"

Comment 7 Mike McCune 2013-06-11 20:03:48 UTC
can we get this re-tested on the latest puddle we built today?  we fixed some pulp and katello memory related issues in some other BZs.

Comment 9 Mike McCune 2013-06-11 20:22:34 UTC
and another fix that may help with this:

https://bugzilla.redhat.com/show_bug.cgi?id=972230

Comment 10 Garik Khachikyan 2013-06-12 07:03:03 UTC
sure thing Mike.

thanks for the links!

Comment 11 Corey Welton 2013-06-12 13:21:44 UTC
I successfully created a CVD, but since we saw this error intermittently, we don't want to close it yet.

Comment 13 Garik Khachikyan 2013-06-13 13:28:30 UTC
good news: works for me now (got ack from Og, Corey too)

I tried making their publish, promoting and consume - so far soo good:)

version that get this fix:
---
candlepin-0.8.9-1.el6_4.noarch
candlepin-cert-consumer-smqa-x3550m3-03.lab.eng.brq.redhat.com-1.0-1.noarch
candlepin-scl-1-5.el6_4.noarch
candlepin-scl-quartz-2.1.5-5.el6_4.noarch
candlepin-scl-rhino-1.7R3-1.el6_4.noarch
candlepin-scl-runtime-1-5.el6_4.noarch
candlepin-selinux-0.8.9-1.el6_4.noarch
candlepin-tomcat6-0.8.9-1.el6_4.noarch
elasticsearch-0.19.9-8.el6sat.noarch
katello-1.4.2-12.el6sat.noarch
katello-agent-1.4.3-1.git.1.24fe511.el6.noarch
katello-all-1.4.2-12.el6sat.noarch
katello-candlepin-cert-key-pair-1.0-1.noarch
katello-certs-tools-1.4.2-2.el6sat.noarch
katello-cli-1.4.2-7.el6sat.noarch
katello-cli-common-1.4.2-7.el6sat.noarch
katello-common-1.4.2-12.el6sat.noarch
katello-configure-1.4.3-15.el6sat.noarch
katello-configure-foreman-1.4.3-15.el6sat.noarch
katello-foreman-all-1.4.2-12.el6sat.noarch
katello-glue-candlepin-1.4.2-12.el6sat.noarch
katello-glue-elasticsearch-1.4.2-12.el6sat.noarch
katello-glue-pulp-1.4.2-12.el6sat.noarch
katello-qpid-broker-key-pair-1.0-1.noarch
katello-qpid-client-key-pair-1.0-1.noarch
katello-selinux-1.4.3-3.el6sat.noarch
m2crypto-0.21.1.pulp-8.el6sat.x86_64
mod_wsgi-3.4-1.pulp.el6sat.x86_64
pulp-rpm-handlers-2.1.1-1.el6sat.noarch
pulp-rpm-plugins-2.1.1-1.el6sat.noarch
pulp-selinux-2.1.1-1.el6sat.noarch
pulp-server-2.1.1-1.el6sat.noarch
python-isodate-0.5.0-1.pulp.el6sat.noarch
python-oauth2-1.5.170-3.pulp.el6sat.noarch
python-pulp-agent-lib-2.1.1-1.el6sat.noarch
python-pulp-common-2.1.1-1.el6sat.noarch
python-pulp-rpm-common-2.1.1-1.el6sat.noarch
python-qpid-0.18-5.el6_4.noarch
python-rhsm-1.8.0-1.pulp.el6sat.x86_64
qpid-cpp-client-0.14-22.el6_3.x86_64
qpid-cpp-client-ssl-0.14-22.el6_3.x86_64
qpid-cpp-server-0.14-22.el6_3.x86_64
qpid-cpp-server-ssl-0.14-22.el6_3.x86_64
ruby193-rubygem-foreman-katello-engine-0.0.8-6.el6sat.noarch
ruby193-rubygem-katello-foreman-engine-0.0.3-4.el6sat.noarch
ruby193-rubygem-katello_api-0.0.3-2.el6_4.noarch
ruby193-rubygem-ldap_fluff-0.2.2-1.el6sat.noarch
signo-katello-0.0.16-1.el6sat.noarch


Note You need to log in before you can comment on or make changes to this bug.