Created attachment 759605 [details] logs Description of problem: My tries publishing content view that contains synced RHEL 6.3 and RHEL 6.4 repos (64bit) always are ending with the "RestClient::RequestTimeout" exception and failing. Version-Release number of selected component (if applicable): candlepin-0.8.9-1.el6_4.noarch candlepin-cert-consumer-sun-x4440-01.rhts.eng.bos.redhat.com-1.0-1.noarch candlepin-scl-1-5.el6_4.noarch candlepin-scl-quartz-2.1.5-5.el6_4.noarch candlepin-scl-rhino-1.7R3-1.el6_4.noarch candlepin-scl-runtime-1-5.el6_4.noarch candlepin-selinux-0.8.9-1.el6_4.noarch candlepin-tomcat6-0.8.9-1.el6_4.noarch elasticsearch-0.19.9-8.el6sat.noarch katello-1.4.2-11.el6sat.noarch katello-agent-1.4.3-1.git.1.24fe511.el6.noarch katello-all-1.4.2-11.el6sat.noarch katello-candlepin-cert-key-pair-1.0-1.noarch katello-certs-tools-1.4.2-2.el6sat.noarch katello-cli-1.4.2-7.el6sat.noarch katello-cli-common-1.4.2-7.el6sat.noarch katello-common-1.4.2-11.el6sat.noarch katello-configure-1.4.3-14.el6sat.noarch katello-configure-foreman-1.4.3-14.el6sat.noarch katello-foreman-all-1.4.2-11.el6sat.noarch katello-glue-candlepin-1.4.2-11.el6sat.noarch katello-glue-elasticsearch-1.4.2-11.el6sat.noarch katello-glue-pulp-1.4.2-11.el6sat.noarch katello-qpid-broker-key-pair-1.0-1.noarch katello-qpid-client-key-pair-1.0-1.noarch katello-selinux-1.4.3-3.el6sat.noarch m2crypto-0.21.1.pulp-8.el6sat.x86_64 mod_wsgi-3.4-1.pulp.el6sat.x86_64 pulp-rpm-handlers-2.1.1-1.el6sat.noarch pulp-rpm-plugins-2.1.1-1.el6sat.noarch pulp-selinux-2.1.1-1.el6sat.noarch pulp-server-2.1.1-1.el6sat.noarch python-isodate-0.5.0-1.pulp.el6sat.noarch python-oauth2-1.5.170-3.pulp.el6sat.noarch python-pulp-agent-lib-2.1.1-1.el6sat.noarch python-pulp-common-2.1.1-1.el6sat.noarch python-pulp-rpm-common-2.1.1-1.el6sat.noarch python-qpid-0.18-5.el6_4.noarch python-rhsm-1.8.0-1.pulp.el6sat.x86_64 qpid-cpp-client-0.14-22.el6_3.x86_64 qpid-cpp-client-ssl-0.14-22.el6_3.x86_64 qpid-cpp-server-0.14-22.el6_3.x86_64 qpid-cpp-server-ssl-0.14-22.el6_3.x86_64 ruby193-rubygem-foreman-katello-engine-0.0.8-5.el6sat.noarch ruby193-rubygem-katello-foreman-engine-0.0.3-4.el6sat.noarch ruby193-rubygem-katello_api-0.0.3-2.el6_4.noarch ruby193-rubygem-ldap_fluff-0.2.2-1.el6sat.noarch signo-katello-0.0.16-1.el6sat.noarch How reproducible: 3 from 3 attempts. Steps to Reproduce (I tried all through UI): 1.import valid manifest 2.enable RHEL6 x86_64 repos (6.1 - 6.4, 6Server) 3.sync RHEL 6.3 and RHEL 6.4 repos 4. make content view definition and add that 2 synced repos 5. try to publish it Actual results: after running about 10 min it fails (see above) Expected results: no failure. Content should be published making it possible to promote to next environments and consume it! Additional info: logs are attached Please not that this could have critical impact on our clients blocking them from consuming Red Hat content.
Created attachment 759606 [details] UI exception
How much RAM does your server have?
12GB and configured via: --- -katello-web-workers=2 --job-workers=2 --es-min-mem=512M --es-max-mem=1024M
Seeing this as well in the latest puddle. [root@mgmt9 ~]# cat /proc/meminfo MemTotal: 8059328 kB MemFree: 439728 kB Buffers: 314148 kB Cached: 2782796 kB SwapCached: 44752 kB Active: 4727948 kB Inactive: 2411756 kB Active(anon): 2932732 kB Inactive(anon): 1119784 kB Active(file): 1795216 kB Inactive(file): 1291972 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 8191992 kB SwapFree: 8070820 kB Dirty: 68 kB Writeback: 0 kB AnonPages: 4019584 kB Mapped: 324644 kB Shmem: 9752 kB Slab: 371520 kB SReclaimable: 297356 kB SUnreclaim: 74164 kB KernelStack: 3328 kB PageTables: 31816 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 12221656 kB Committed_AS: 3762992 kB VmallocTotal: 34359738367 kB VmallocUsed: 28180 kB VmallocChunk: 34359700412 kB HardwareCorrupted: 0 kB AnonHugePages: 1611776 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 9600 kB DirectMap2M: 8378368 kB
FWIW, after a third attempt to publish my CVD, I get the following error in the UI Pulp has already been taken katello production log: [ERROR 2013-06-11 10:12:33 pulp_rest #22431] RestClient.post "https://mgmt9.rhq.lab.eng.bos.redhat.com/pulp/api/v2/content/units/rpm/search/", 8142 byte(s) length, "Accept"=>"application/json", "Accept-Encoding"=>"gzip, deflate", "Authorization"=>"OAuth oauth_body_hash=\"2jmj7l5rSw0yVb%2FvlWAYkK%2FYBwk%3D\", oau th_consumer_key=\"katello\", oauth_nonce=\"7tf5jJZrVdsYk0v7GcgD4aazr0Pn9cl3yC6PpzBS3g\", oauth_signature=\"2LYObsRBvQ1UeATHEmxFbeg6eCs%3D\", oauth_signature_ method=\"HMAC-SHA1\", oauth_timestamp=\"1370959893\", oauth_version=\"1.0\"", "Content-Length"=>"8142", "Content-Type"=>"application/json", "pulp-user"=>"adm in" "production.log" 506L, 53313C | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:78:in `call' | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:78:in `block in add' | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `call' | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:65:in `execute' | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/lifecycle.rb:38:in `run_callbacks' | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/worker.rb:127:in `start' | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:101:in `run' | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:89:in `block in run_process' | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/application.rb:249:in `call' | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/application.rb:249:in `block in start_proc' | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/daemonize.rb:197:in `call' | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/daemonize.rb:197:in `call_as_daemon' | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/application.rb:253:in `start_proc' | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/application.rb:293:in `start' | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/controller.rb:70:in `run' | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons.rb:195:in `block in run_proc' | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/cmdline.rb:109:in `call' | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons/cmdline.rb:109:in `catch_exceptions' | /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.1.4/lib/daemons.rb:194:in `run_proc' | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:87:in `run_process' | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:80:in `block in daemonize' | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:78:in `times' | /opt/rh/ruby193/root/usr/share/gems/gems/delayed_job-3.0.2/lib/delayed/command.rb:78:in `daemonize' | script/delayed_job:5:in `<main>'
https://bugzilla.redhat.com/show_bug.cgi?id=972852 is to track "Pulp has been taken"
can we get this re-tested on the latest puddle we built today? we fixed some pulp and katello memory related issues in some other BZs.
https://bugzilla.redhat.com/show_bug.cgi?id=960278
and another fix that may help with this: https://bugzilla.redhat.com/show_bug.cgi?id=972230
sure thing Mike. thanks for the links!
I successfully created a CVD, but since we saw this error intermittently, we don't want to close it yet.
good news: works for me now (got ack from Og, Corey too) I tried making their publish, promoting and consume - so far soo good:) version that get this fix: --- candlepin-0.8.9-1.el6_4.noarch candlepin-cert-consumer-smqa-x3550m3-03.lab.eng.brq.redhat.com-1.0-1.noarch candlepin-scl-1-5.el6_4.noarch candlepin-scl-quartz-2.1.5-5.el6_4.noarch candlepin-scl-rhino-1.7R3-1.el6_4.noarch candlepin-scl-runtime-1-5.el6_4.noarch candlepin-selinux-0.8.9-1.el6_4.noarch candlepin-tomcat6-0.8.9-1.el6_4.noarch elasticsearch-0.19.9-8.el6sat.noarch katello-1.4.2-12.el6sat.noarch katello-agent-1.4.3-1.git.1.24fe511.el6.noarch katello-all-1.4.2-12.el6sat.noarch katello-candlepin-cert-key-pair-1.0-1.noarch katello-certs-tools-1.4.2-2.el6sat.noarch katello-cli-1.4.2-7.el6sat.noarch katello-cli-common-1.4.2-7.el6sat.noarch katello-common-1.4.2-12.el6sat.noarch katello-configure-1.4.3-15.el6sat.noarch katello-configure-foreman-1.4.3-15.el6sat.noarch katello-foreman-all-1.4.2-12.el6sat.noarch katello-glue-candlepin-1.4.2-12.el6sat.noarch katello-glue-elasticsearch-1.4.2-12.el6sat.noarch katello-glue-pulp-1.4.2-12.el6sat.noarch katello-qpid-broker-key-pair-1.0-1.noarch katello-qpid-client-key-pair-1.0-1.noarch katello-selinux-1.4.3-3.el6sat.noarch m2crypto-0.21.1.pulp-8.el6sat.x86_64 mod_wsgi-3.4-1.pulp.el6sat.x86_64 pulp-rpm-handlers-2.1.1-1.el6sat.noarch pulp-rpm-plugins-2.1.1-1.el6sat.noarch pulp-selinux-2.1.1-1.el6sat.noarch pulp-server-2.1.1-1.el6sat.noarch python-isodate-0.5.0-1.pulp.el6sat.noarch python-oauth2-1.5.170-3.pulp.el6sat.noarch python-pulp-agent-lib-2.1.1-1.el6sat.noarch python-pulp-common-2.1.1-1.el6sat.noarch python-pulp-rpm-common-2.1.1-1.el6sat.noarch python-qpid-0.18-5.el6_4.noarch python-rhsm-1.8.0-1.pulp.el6sat.x86_64 qpid-cpp-client-0.14-22.el6_3.x86_64 qpid-cpp-client-ssl-0.14-22.el6_3.x86_64 qpid-cpp-server-0.14-22.el6_3.x86_64 qpid-cpp-server-ssl-0.14-22.el6_3.x86_64 ruby193-rubygem-foreman-katello-engine-0.0.8-6.el6sat.noarch ruby193-rubygem-katello-foreman-engine-0.0.3-4.el6sat.noarch ruby193-rubygem-katello_api-0.0.3-2.el6_4.noarch ruby193-rubygem-ldap_fluff-0.2.2-1.el6sat.noarch signo-katello-0.0.16-1.el6sat.noarch