Bug 973181 - RestClient::RequestTimeout on publishing content view
RestClient::RequestTimeout on publishing content view
Status: CLOSED WORKSFORME
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Content Management (Show other bugs)
6.0.1
Unspecified Unspecified
high Severity unspecified (vote)
: Unspecified
: --
Assigned To: Katello Bug Bin
Og Maciel
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-11 08:34 EDT by Garik Khachikyan
Modified: 2015-01-04 16:59 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-13 09:28:30 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
logs (2.72 MB, application/gzip)
2013-06-11 08:34 EDT, Garik Khachikyan
no flags Details
UI exception (47.13 KB, image/png)
2013-06-11 08:41 EDT, Garik Khachikyan
no flags Details

  None (edit)
Description Garik Khachikyan 2013-06-11 08:34:55 EDT
Created attachment 759605 [details]
logs

Description of problem:
My tries publishing content view that contains synced RHEL 6.3 and RHEL 6.4 repos (64bit) always are ending with the "RestClient::RequestTimeout" exception and failing.

Version-Release number of selected component (if applicable):
candlepin-0.8.9-1.el6_4.noarch
candlepin-cert-consumer-sun-x4440-01.rhts.eng.bos.redhat.com-1.0-1.noarch
candlepin-scl-1-5.el6_4.noarch
candlepin-scl-quartz-2.1.5-5.el6_4.noarch
candlepin-scl-rhino-1.7R3-1.el6_4.noarch
candlepin-scl-runtime-1-5.el6_4.noarch
candlepin-selinux-0.8.9-1.el6_4.noarch
candlepin-tomcat6-0.8.9-1.el6_4.noarch
elasticsearch-0.19.9-8.el6sat.noarch
katello-1.4.2-11.el6sat.noarch
katello-agent-1.4.3-1.git.1.24fe511.el6.noarch
katello-all-1.4.2-11.el6sat.noarch
katello-candlepin-cert-key-pair-1.0-1.noarch
katello-certs-tools-1.4.2-2.el6sat.noarch
katello-cli-1.4.2-7.el6sat.noarch
katello-cli-common-1.4.2-7.el6sat.noarch
katello-common-1.4.2-11.el6sat.noarch
katello-configure-1.4.3-14.el6sat.noarch
katello-configure-foreman-1.4.3-14.el6sat.noarch
katello-foreman-all-1.4.2-11.el6sat.noarch
katello-glue-candlepin-1.4.2-11.el6sat.noarch
katello-glue-elasticsearch-1.4.2-11.el6sat.noarch
katello-glue-pulp-1.4.2-11.el6sat.noarch
katello-qpid-broker-key-pair-1.0-1.noarch
katello-qpid-client-key-pair-1.0-1.noarch
katello-selinux-1.4.3-3.el6sat.noarch
m2crypto-0.21.1.pulp-8.el6sat.x86_64
mod_wsgi-3.4-1.pulp.el6sat.x86_64
pulp-rpm-handlers-2.1.1-1.el6sat.noarch
pulp-rpm-plugins-2.1.1-1.el6sat.noarch
pulp-selinux-2.1.1-1.el6sat.noarch
pulp-server-2.1.1-1.el6sat.noarch
python-isodate-0.5.0-1.pulp.el6sat.noarch
python-oauth2-1.5.170-3.pulp.el6sat.noarch
python-pulp-agent-lib-2.1.1-1.el6sat.noarch
python-pulp-common-2.1.1-1.el6sat.noarch
python-pulp-rpm-common-2.1.1-1.el6sat.noarch
python-qpid-0.18-5.el6_4.noarch
python-rhsm-1.8.0-1.pulp.el6sat.x86_64
qpid-cpp-client-0.14-22.el6_3.x86_64
qpid-cpp-client-ssl-0.14-22.el6_3.x86_64
qpid-cpp-server-0.14-22.el6_3.x86_64
qpid-cpp-server-ssl-0.14-22.el6_3.x86_64
ruby193-rubygem-foreman-katello-engine-0.0.8-5.el6sat.noarch
ruby193-rubygem-katello-foreman-engine-0.0.3-4.el6sat.noarch
ruby193-rubygem-katello_api-0.0.3-2.el6_4.noarch
ruby193-rubygem-ldap_fluff-0.2.2-1.el6sat.noarch
signo-katello-0.0.16-1.el6sat.noarch

How reproducible:
3 from 3 attempts.

Steps to Reproduce (I tried all through UI):
1.import valid manifest
2.enable RHEL6 x86_64 repos (6.1 - 6.4, 6Server)
3.sync RHEL 6.3 and RHEL 6.4 repos
4. make content view definition and add that 2 synced repos
5. try to publish it

Actual results:
after running about 10 min it fails (see above)

Expected results:
no failure. Content should be published making it possible to promote to next environments and consume it!

Additional info:
logs are attached

Please not that this could have critical impact on our clients blocking them from consuming Red Hat content.
Comment 1 Garik Khachikyan 2013-06-11 08:41:18 EDT
Created attachment 759606 [details]
UI exception
Comment 2 Og Maciel 2013-06-11 09:17:03 EDT
How much RAM does your server have?
Comment 3 Garik Khachikyan 2013-06-11 09:20:41 EDT
12GB and configured via:
---
-katello-web-workers=2 --job-workers=2 --es-min-mem=512M --es-max-mem=1024M
Comment 4 Corey Welton 2013-06-11 10:17:00 EDT
Seeing this as well in the latest puddle.

[root@mgmt9 ~]# cat /proc/meminfo
MemTotal:        8059328 kB
MemFree:          439728 kB
Buffers:          314148 kB
Cached:          2782796 kB
SwapCached:        44752 kB
Active:          4727948 kB
Inactive:        2411756 kB
Active(anon):    2932732 kB
Inactive(anon):  1119784 kB
Active(file):    1795216 kB
Inactive(file):  1291972 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       8191992 kB
SwapFree:        8070820 kB
Dirty:                68 kB
Writeback:             0 kB
AnonPages:       4019584 kB
Mapped:           324644 kB
Shmem:              9752 kB
Slab:             371520 kB
SReclaimable:     297356 kB
SUnreclaim:        74164 kB
KernelStack:        3328 kB
PageTables:        31816 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    12221656 kB
Committed_AS:    3762992 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       28180 kB
VmallocChunk:   34359700412 kB
HardwareCorrupted:     0 kB
AnonHugePages:   1611776 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        9600 kB
DirectMap2M:     8378368 kB
Comment 5 Corey Welton 2013-06-11 11:28:07 EDT
FWIW, after a third attempt to publish my CVD, I get the following error in the UI

Pulp has already been taken


katello production log:

[ERROR 2013-06-11 10:12:33 pulp_rest  #22431] RestClient.post "https://mgmt9.rhq.lab.eng.bos.redhat.com/pulp/api/v2/content/units/rpm/search/", 8142 byte(s) 
length, "Accept"=>"application/json", "Accept-Encoding"=>"gzip, deflate", "Authorization"=>"OAuth oauth_body_hash=\"2jmj7l5rSw0yVb%2FvlWAYkK%2FYBwk%3D\", oau
th_consumer_key=\"katello\", oauth_nonce=\"7tf5jJZrVdsYk0v7GcgD4aazr0Pn9cl3yC6PpzBS3g\", oauth_signature=\"2LYObsRBvQ1UeATHEmxFbeg6eCs%3D\", oauth_signature_
method=\"HMAC-SHA1\", oauth_timestamp=\"1370959893\", oauth_version=\"1.0\"", "Content-Length"=>"8142", "Content-Type"=>"application/json", "pulp-user"=>"adm
in"
"production.log" 506L, 53313C
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:78:in `call'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:78:in `block in add'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `call'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `execute'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:38:in `run_callbacks'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:127:in `start'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:101:in `run'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:89:in `block in run_process'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/application.rb:249:in `call'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/application.rb:249:in `block in start_proc'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/daemonize.rb:197:in `call'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/daemonize.rb:197:in `call_as_daemon'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/application.rb:253:in `start_proc'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/application.rb:293:in `start'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/controller.rb:70:in `run'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons.rb:195:in `block in run_proc'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/cmdline.rb:109:in `call'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/cmdline.rb:109:in `catch_exceptions'
 | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons.rb:194:in `run_proc'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:87:in `run_process'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:80:in `block in daemonize'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:78:in `times'
 | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:78:in `daemonize'
 | script/delayed_job:5:in `<main>'
Comment 6 Og Maciel 2013-06-11 11:46:50 EDT
https://bugzilla.redhat.com/show_bug.cgi?id=972852 is to track "Pulp has been taken"
Comment 7 Mike McCune 2013-06-11 16:03:48 EDT
can we get this re-tested on the latest puddle we built today?  we fixed some pulp and katello memory related issues in some other BZs.
Comment 9 Mike McCune 2013-06-11 16:22:34 EDT
and another fix that may help with this:

https://bugzilla.redhat.com/show_bug.cgi?id=972230
Comment 10 Garik Khachikyan 2013-06-12 03:03:03 EDT
sure thing Mike.

thanks for the links!
Comment 11 Corey Welton 2013-06-12 09:21:44 EDT
I successfully created a CVD, but since we saw this error intermittently, we don't want to close it yet.
Comment 13 Garik Khachikyan 2013-06-13 09:28:30 EDT
good news: works for me now (got ack from Og, Corey too)

I tried making their publish, promoting and consume - so far soo good:)

version that get this fix:
---
candlepin-0.8.9-1.el6_4.noarch
candlepin-cert-consumer-smqa-x3550m3-03.lab.eng.brq.redhat.com-1.0-1.noarch
candlepin-scl-1-5.el6_4.noarch
candlepin-scl-quartz-2.1.5-5.el6_4.noarch
candlepin-scl-rhino-1.7R3-1.el6_4.noarch
candlepin-scl-runtime-1-5.el6_4.noarch
candlepin-selinux-0.8.9-1.el6_4.noarch
candlepin-tomcat6-0.8.9-1.el6_4.noarch
elasticsearch-0.19.9-8.el6sat.noarch
katello-1.4.2-12.el6sat.noarch
katello-agent-1.4.3-1.git.1.24fe511.el6.noarch
katello-all-1.4.2-12.el6sat.noarch
katello-candlepin-cert-key-pair-1.0-1.noarch
katello-certs-tools-1.4.2-2.el6sat.noarch
katello-cli-1.4.2-7.el6sat.noarch
katello-cli-common-1.4.2-7.el6sat.noarch
katello-common-1.4.2-12.el6sat.noarch
katello-configure-1.4.3-15.el6sat.noarch
katello-configure-foreman-1.4.3-15.el6sat.noarch
katello-foreman-all-1.4.2-12.el6sat.noarch
katello-glue-candlepin-1.4.2-12.el6sat.noarch
katello-glue-elasticsearch-1.4.2-12.el6sat.noarch
katello-glue-pulp-1.4.2-12.el6sat.noarch
katello-qpid-broker-key-pair-1.0-1.noarch
katello-qpid-client-key-pair-1.0-1.noarch
katello-selinux-1.4.3-3.el6sat.noarch
m2crypto-0.21.1.pulp-8.el6sat.x86_64
mod_wsgi-3.4-1.pulp.el6sat.x86_64
pulp-rpm-handlers-2.1.1-1.el6sat.noarch
pulp-rpm-plugins-2.1.1-1.el6sat.noarch
pulp-selinux-2.1.1-1.el6sat.noarch
pulp-server-2.1.1-1.el6sat.noarch
python-isodate-0.5.0-1.pulp.el6sat.noarch
python-oauth2-1.5.170-3.pulp.el6sat.noarch
python-pulp-agent-lib-2.1.1-1.el6sat.noarch
python-pulp-common-2.1.1-1.el6sat.noarch
python-pulp-rpm-common-2.1.1-1.el6sat.noarch
python-qpid-0.18-5.el6_4.noarch
python-rhsm-1.8.0-1.pulp.el6sat.x86_64
qpid-cpp-client-0.14-22.el6_3.x86_64
qpid-cpp-client-ssl-0.14-22.el6_3.x86_64
qpid-cpp-server-0.14-22.el6_3.x86_64
qpid-cpp-server-ssl-0.14-22.el6_3.x86_64
ruby193-rubygem-foreman-katello-engine-0.0.8-6.el6sat.noarch
ruby193-rubygem-katello-foreman-engine-0.0.3-4.el6sat.noarch
ruby193-rubygem-katello_api-0.0.3-2.el6_4.noarch
ruby193-rubygem-ldap_fluff-0.2.2-1.el6sat.noarch
signo-katello-0.0.16-1.el6sat.noarch

Note You need to log in before you can comment on or make changes to this bug.