Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 895814 Details for
Bug 1088481
qpidd restart needed to unblock celery processing
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
Screen log from another node
screen.log (text/plain), 24.90 KB, created by
mkovacik
on 2014-05-15 09:08:18 UTC
(
hide
)
Description:
Screen log from another node
Filename:
MIME Type:
Creator:
mkovacik
Created:
2014-05-15 09:08:18 UTC
Size:
24.90 KB
patch
obsolete
>[root@ec2-54-73-155-135 pulp_auto]# qpid-stat -q >Queues > queue dur autoDel excl msg msgIn msgOut bytes bytesIn bytesOut cons bind > ============================================================================================================================================================================ > 0c483076-1dca-4613-b53f-ff4a2a7d83c7:1.0 Y Y 0 9 9 0 3.96k 3.96k 1 2 > 2b4bf97f-ed2b-46a7-996c-5efad6241937:1.0 Y Y 0 10 10 0 4.05k 4.05k 1 2 > 3a306591-5e58-49d2-871c-2b05f524e444:1.0 Y Y 0 8 8 0 3.83k 3.83k 1 2 > 3a306591-5e58-49d2-871c-2b05f524e444:3.0 Y Y 0 1.23k 1.23k 0 120k 120k 1 2 > 3eba8fd4-bb7d-44b3-9577-789f9a74d4fe:1.0 Y Y 0 2 2 0 172 172 1 2 > 4083d290-92f1-47bf-83c3-0209e00e43f1:1.0 Y Y 0 2.75k 2.75k 0 1.21m 1.21m 1 2 > 465bd5e4-b9f9-41a7-857c-d838937add4a:1.0 Y Y 0 4 4 0 1.83k 1.83k 1 2 > 4f7b9644-596a-418d-8abf-81c000eaf2dd:1.0 Y Y 0 2 2 0 172 172 1 2 > 53336b36-a3df-4551-92c1-88b0692b7137:1.0 Y Y 0 0 0 0 0 0 1 2 > 54fa1223-a798-4a2f-a98d-374ffd92b3e5:1.0 Y Y 0 4 4 0 1.77k 1.77k 1 2 > 5da1bf0f-110a-4515-bb4f-072ad4904c17:1.0 Y Y 0 2.75k 2.75k 0 1.21m 1.21m 1 2 > 6fc9339f-a70a-4b94-972f-55021dfddd8b:1.0 Y Y 0 9 9 0 3.96k 3.96k 1 2 > 732282ee-fa0b-42fc-af5a-001f43d473c1:1.0 Y Y 0 2 2 0 172 172 1 2 > 801f499b-7150-4aba-992f-af5522ef5013:1.0 Y Y 0 8 8 0 3.83k 3.83k 1 2 > 801f499b-7150-4aba-992f-af5522ef5013:3.0 Y Y 0 1.23k 1.23k 0 120k 120k 1 2 > 81054d92-e2e0-47ae-af35-a9bbf2c638cb:1.0 Y Y 0 2 2 0 172 172 1 2 > 813ea04f-3984-4c56-ba9e-c64b69097c68:1.0 Y Y 0 2 2 0 172 172 1 2 > 84f7124e-10b3-430e-a983-6c262308ff98:1.0 Y Y 0 10 10 0 4.05k 4.05k 1 2 > 920b0e9d-cb20-46c5-a4a9-366bd7a966e7:1.0 Y Y 0 2.74k 2.74k 0 1.21m 1.21m 1 2 > 93b3abba-8df0-421a-8b3c-b14ff22bb339:1.0 Y Y 0 2.74k 2.74k 0 1.20m 1.20m 1 2 > a0605a68-b7e8-45ef-9dd8-776496264385:1.0 Y Y 0 8 8 0 3.83k 3.83k 1 2 > a0605a68-b7e8-45ef-9dd8-776496264385:3.0 Y Y 0 1.23k 1.23k 0 120k 120k 1 2 > bcbd7ced-d490-4653-b0e0-493f67d0d8f5:0.0 Y Y 0 0 0 0 0 0 1 2 > c85f75c0-1f27-48a2-aee3-4ef9dbe64cf8:1.0 Y Y 0 4 4 0 1.81k 1.81k 1 2 > cb62863b-fd56-4203-9277-4b299d773d5a:1.0 Y Y 0 8 8 0 3.83k 3.83k 1 2 > cb62863b-fd56-4203-9277-4b299d773d5a:3.0 Y Y 0 1.23k 1.23k 0 120k 120k 1 2 > celery Y 0 1.53k 1.53k 0 1.05m 1.05m 4 2 > celeryev.3cb895b1-a823-4ded-bddd-04dbf9d61e20 Y 130k 151k 20.5k 104m 121m 16.4m 1 2 > celeryev.606350b0-f95e-461e-98d8-72c4d6370c49 Y 130k 151k 20.5k 104m 121m 16.4m 1 2 > d15abb24-6339-40e0-bf1b-0a463654b066:1.0 Y Y 0 0 0 0 0 0 1 2 > e107fd36-baf1-4385-b4cc-8eece2e20c44:1.0 Y Y 0 9 9 0 3.96k 3.96k 1 2 > f8861cea-0b01-4e1e-b25b-31e767c80762:1.0 Y Y 0 4 4 0 1.81k 1.81k 1 2 > f8861cea-0b01-4e1e-b25b-31e767c80762:3.0 Y Y 0 1.23k 1.23k 0 120k 120k 1 2 > fcdcd799-c378-4c1b-8400-5d662642513b:1.0 Y Y 0 4 4 0 1.83k 1.83k 1 2 > pulp.agent.my_consumer Y 0 52 52 0 52.7k 52.7k 0 1 > pulp.task Y 0 1.14k 1.14k 0 1.11m 1.11m 3 1 > reserved_resource_worker-0@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com Y 0 616 616 0 592k 592k 1 2 > reserved_resource_worker-0@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com.celery.pidbox Y 1.23k 1.23k 0 718k 718k 0 1 2 > reserved_resource_worker-1@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com Y 0 18 18 0 17.1k 17.1k 1 2 > reserved_resource_worker-1@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com.celery.pidbox Y 1.23k 1.23k 0 718k 718k 0 1 2 > reserved_resource_worker-2@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com Y 0 30 30 0 27.7k 27.7k 1 2 > reserved_resource_worker-2@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com.celery.pidbox Y 1.23k 1.23k 0 718k 718k 0 1 2 > reserved_resource_worker-3@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com Y 0 1.84k 1.84k 0 1.79m 1.79m 1 2 > reserved_resource_worker-3@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com.celery.pidbox Y 1.23k 1.23k 0 718k 718k 0 1 2 > resource_manager Y 0 2.56k 2.56k 0 1.96m 1.96m 1 2 > resource_manager@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com.celery.pidbox Y 1.23k 1.23k 0 718k 718k 0 1 2 >[root@ec2-54-73-155-135 pulp_auto]# qpid-config queues >Queue Name Attributes >=================================================================================================================== >0c483076-1dca-4613-b53f-ff4a2a7d83c7:1.0 auto-del excl >126f10e2-4fdc-3d9f-8820-d47363d72b4d.reply.celery.pidbox auto-del --limit-policy=ring --argument passive=False --argument exclusive=False --argument qpid.auto_delete_timeout=3 --argument arguments={u'x-expires': 10000} >2b4bf97f-ed2b-46a7-996c-5efad6241937:1.0 auto-del excl >3a306591-5e58-49d2-871c-2b05f524e444:1.0 auto-del excl >3a306591-5e58-49d2-871c-2b05f524e444:3.0 auto-del excl >3eba8fd4-bb7d-44b3-9577-789f9a74d4fe:1.0 auto-del excl >4083d290-92f1-47bf-83c3-0209e00e43f1:1.0 auto-del excl >465bd5e4-b9f9-41a7-857c-d838937add4a:1.0 auto-del excl >4f7b9644-596a-418d-8abf-81c000eaf2dd:1.0 auto-del excl >53336b36-a3df-4551-92c1-88b0692b7137:1.0 auto-del excl >54fa1223-a798-4a2f-a98d-374ffd92b3e5:1.0 auto-del excl >5bc4e8e8-c752-4006-a87a-621fc6bb8efc:0.0 auto-del excl >5da1bf0f-110a-4515-bb4f-072ad4904c17:1.0 auto-del excl >6fc9339f-a70a-4b94-972f-55021dfddd8b:1.0 auto-del excl >732282ee-fa0b-42fc-af5a-001f43d473c1:1.0 auto-del excl >801f499b-7150-4aba-992f-af5522ef5013:1.0 auto-del excl >801f499b-7150-4aba-992f-af5522ef5013:3.0 auto-del excl >81054d92-e2e0-47ae-af35-a9bbf2c638cb:1.0 auto-del excl >813ea04f-3984-4c56-ba9e-c64b69097c68:1.0 auto-del excl >84f7124e-10b3-430e-a983-6c262308ff98:1.0 auto-del excl >920b0e9d-cb20-46c5-a4a9-366bd7a966e7:1.0 auto-del excl >93b3abba-8df0-421a-8b3c-b14ff22bb339:1.0 auto-del excl >a0605a68-b7e8-45ef-9dd8-776496264385:1.0 auto-del excl >a0605a68-b7e8-45ef-9dd8-776496264385:3.0 auto-del excl >c85f75c0-1f27-48a2-aee3-4ef9dbe64cf8:1.0 auto-del excl >cb62863b-fd56-4203-9277-4b299d773d5a:1.0 auto-del excl >cb62863b-fd56-4203-9277-4b299d773d5a:3.0 auto-del excl >celery --durable --argument passive=False --argument exclusive=False --argument qpid.auto_delete_timeout=3 --argument arguments=None >celeryev.3cb895b1-a823-4ded-bddd-04dbf9d61e20 auto-del --limit-policy=ring --argument passive=False --argument exclusive=False --argument qpid.auto_delete_timeout=3 --argument arguments={} >celeryev.606350b0-f95e-461e-98d8-72c4d6370c49 auto-del --limit-policy=ring --argument passive=False --argument exclusive=False --argument qpid.auto_delete_timeout=3 --argument arguments={} >d15abb24-6339-40e0-bf1b-0a463654b066:1.0 auto-del excl >e107fd36-baf1-4385-b4cc-8eece2e20c44:1.0 auto-del excl >f8861cea-0b01-4e1e-b25b-31e767c80762:1.0 auto-del excl >f8861cea-0b01-4e1e-b25b-31e767c80762:3.0 auto-del excl >fcdcd799-c378-4c1b-8400-5d662642513b:1.0 auto-del excl >pulp.agent.my_consumer --durable >pulp.task --durable >reserved_resource_worker-0@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com --durable --argument passive=False --argument exclusive=False --argument qpid.auto_delete_timeout=3 --argument arguments=None >reserved_resource_worker-0@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com.celery.pidbox auto-del --limit-policy=ring --argument passive=False --argument exclusive=False --argument qpid.auto_delete_timeout=3 --argument arguments=None >reserved_resource_worker-1@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com --durable --argument passive=False --argument exclusive=False --argument qpid.auto_delete_timeout=3 --argument arguments=None >reserved_resource_worker-1@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com.celery.pidbox auto-del --limit-policy=ring --argument passive=False --argument exclusive=False --argument qpid.auto_delete_timeout=3 --argument arguments=None >reserved_resource_worker-2@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com --durable --argument passive=False --argument exclusive=False --argument qpid.auto_delete_timeout=3 --argument arguments=None >reserved_resource_worker-2@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com.celery.pidbox auto-del --limit-policy=ring --argument passive=False --argument exclusive=False --argument qpid.auto_delete_timeout=3 --argument arguments=None >reserved_resource_worker-3@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com --durable --argument passive=False --argument exclusive=False --argument qpid.auto_delete_timeout=3 --argument arguments=None >reserved_resource_worker-3@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com.celery.pidbox auto-del --limit-policy=ring --argument passive=False --argument exclusive=False --argument qpid.auto_delete_timeout=3 --argument arguments=None >resource_manager --durable --argument passive=False --argument exclusive=False --argument qpid.auto_delete_timeout=3 --argument arguments=None >resource_manager@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com.celery.pidbox auto-del --limit-policy=ring --argument passive=False --argument exclusive=False --argument qpid.auto_delete_timeout=3 --argument arguments=None > >### the nosetests command got stuck 2 days ago >[root@ec2-54-73-155-135 pulp_auto]# ps -ef | grep [t]est >root 17137 26502 0 May13 ? 00:00:02 /bin/python /usr/bin/nosetests -v --with-xunit --xunit-file=xunit-_r-1_w-1_b-100000_t-0.4.xml >root 26502 1 0 May13 ? 00:00:00 /bin/bash test.sh >[root@ec2-54-73-155-135 pulp_auto]# date >Thu May 15 08:53:23 UTC 2014 >[root@ec2-54-73-155-135 pulp_auto]# ls -ltrh /usr/share/pulp_auto/*.log | tail -1 >-rw-r--r--. 1 root root 0 May 13 19:18 /usr/share/pulp_auto/xunit-_r-1_w-1_b-100000_t-0.4.xml.log > >### restarting qpidd and pulp services to empty the queues >[root@ec2-54-73-155-135 pulp_auto]# systemctl restart qpidd ># blocked pulp services are hard to restart >[root@ec2-54-73-155-135 pulp_auto]# systemctl restart `systemctl list-unit-files | egrep 'pulp|httpd|goferd' | cut -d\ -f1` >Job for pulp_worker-3.service canceled. > > > > >^Z >[1]+ Stopped systemctl restart `systemctl list-unit-files | egrep 'pulp|httpd|goferd' | cut -d\ -f1` >[root@ec2-54-73-155-135 pulp_auto]# jobs >[1]+ Stopped systemctl restart `systemctl list-unit-files | egrep 'pulp|httpd|goferd' | cut -d\ -f1` >[root@ec2-54-73-155-135 pulp_auto]# jobs -l >[1]+ 20034 Stopped systemctl restart `systemctl list-unit-files | egrep 'pulp|httpd|goferd' | cut -d\ -f1` > >[root@ec2-54-73-155-135 pulp_auto]# date >Thu May 15 08:59:33 UTC 2014 >[1]+ Exit 1 systemctl restart `systemctl list-unit-files | egrep 'pulp|httpd|goferd' | cut -d\ -f1` >[root@ec2-54-73-155-135 pulp_auto]# systemctl stop `systemctl list-unit-files | egrep 'pulp|httpd|goferd' | cut -d\ -f1` >[root@ec2-54-73-155-135 pulp_auto]# systemctl start `systemctl list-unit-files | egrep 'pulp|httpd|goferd' | cut -d\ -f1 > >### assert queues are empty again >[root@ec2-54-73-155-135 pulp_auto]# qpid-stat -q >Queues > queue dur autoDel excl msg msgIn msgOut bytes bytesIn bytesOut cons bind > =========================================================================================================================================================================== > 0ccae12b-4360-40eb-9a9b-4be286be6a74:1.0 Y Y 0 2 2 0 172 172 1 2 > 0e12ad7a-750e-4abf-baeb-ec6b6ad83bc1:1.0 Y Y 0 0 0 0 0 0 1 2 > 1211ae16-cd65-47e3-a8ee-4381f8ac29f7:0.0 Y Y 0 0 0 0 0 0 1 2 > 1384695e-ae94-4472-a78d-093216bed16f:1.0 Y Y 0 4 4 0 1.83k 1.83k 1 2 > 1e400dbe-6047-4687-b0a5-ca892e3ada4d:1.0 Y Y 0 2 2 0 172 172 1 2 > 25a007b7-d2ba-4c7e-adef-009701c9de11:1.0 Y Y 0 17 17 0 6.14k 6.14k 1 2 > 487b5542-9c82-4c45-890b-2eecfc4135df:1.0 Y Y 0 4 4 0 1.81k 1.81k 1 2 > 487b5542-9c82-4c45-890b-2eecfc4135df:3.0 Y Y 0 10 10 0 2.48k 2.48k 1 2 > 564e7cc2-fefa-48ca-b767-c10ef0a1bfd9:1.0 Y Y 0 11 11 0 4.14k 4.14k 1 2 > 625d9ec1-ab85-45c5-9928-42bf8f4c1b78:1.0 Y Y 0 8 8 0 3.83k 3.83k 1 2 > 625d9ec1-ab85-45c5-9928-42bf8f4c1b78:3.0 Y Y 0 9 9 0 2.40k 2.40k 1 2 > 6754ab02-bd8e-4e0e-881d-83ee615c0a03:1.0 Y Y 0 2 2 0 172 172 1 2 > 6fddbc9f-c193-4430-9d8c-f1e5943db4c1:1.0 Y Y 0 0 0 0 0 0 1 2 > 702329c2-ac78-4148-bce6-2cc9d38691b8:1.0 Y Y 0 2 2 0 172 172 1 2 > 92a3161b-c7b3-4c7f-b1ec-9c4b90bccd95:1.0 Y Y 0 9 9 0 3.96k 3.96k 1 2 > a5cde59f-5fb2-466a-9651-3e4ebbfa7485:1.0 Y Y 0 8 8 0 3.83k 3.83k 1 2 > a5cde59f-5fb2-466a-9651-3e4ebbfa7485:3.0 Y Y 0 10 10 0 2.50k 2.50k 1 2 > a66f0b1a-acd8-4acc-927c-04b845463860:1.0 Y Y 0 10 10 0 4.05k 4.05k 1 2 > b649017d-c195-3fff-b37d-b69362a7b438.reply.celery.pidbox Y 5 5 0 4.73k 4.73k 0 1 2 > bb329286-fb6b-40db-97d8-bc31caa87793:1.0 Y Y 0 4 4 0 1.83k 1.83k 1 2 > bed7412b-139c-410a-80cc-cd3711f9b9ab:1.0 Y Y 0 8 8 0 3.83k 3.83k 1 2 > bed7412b-139c-410a-80cc-cd3711f9b9ab:3.0 Y Y 0 7 7 0 2.21k 2.21k 1 2 > bfa538dd-52f5-4afd-af27-29a8f1d9132e:1.0 Y Y 0 2 2 0 172 172 1 2 > celery Y 0 4 4 0 2.85k 2.85k 4 2 > celeryev.cb2651b7-dfe0-4de6-9371-dc4b7aba987d Y 23 23 0 18.0k 18.0k 0 1 2 > celeryev.dc3ca4ad-5998-43ed-973f-49ab81dcc920 Y 24 24 0 18.8k 18.8k 0 1 2 > d3325fc6-6d3c-4664-8420-52b9ce52b129:1.0 Y Y 0 9 9 0 3.96k 3.96k 1 2 > d38e0ad2-d6a4-4fe9-8809-821e8b249cd9:1.0 Y Y 0 8 8 0 3.83k 3.83k 1 2 > d38e0ad2-d6a4-4fe9-8809-821e8b249cd9:3.0 Y Y 0 10 10 0 2.50k 2.50k 1 2 > e4492165-a2c3-4781-b2ef-8f0b4754dcf5:1.0 Y Y 0 9 9 0 3.96k 3.96k 1 2 > pulp.agent.my_consumer Y 0 0 0 0 0 0 0 1 > pulp.task Y 0 0 0 0 0 0 1 1 > reserved_resource_worker-0@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com Y 0 2 2 0 1.77k 1.77k 1 2 > reserved_resource_worker-0@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com.celery.pidbox Y 10 10 0 6.55k 6.55k 0 1 2 > reserved_resource_worker-1@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com Y 0 0 0 0 0 0 1 2 > reserved_resource_worker-1@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com.celery.pidbox Y 12 12 0 7.72k 7.72k 0 1 2 > reserved_resource_worker-2@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com Y 0 0 0 0 0 0 1 2 > reserved_resource_worker-2@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com.celery.pidbox Y 13 13 0 8.30k 8.30k 0 1 2 > reserved_resource_worker-3@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com Y 0 12 12 0 11.1k 11.1k 1 2 > reserved_resource_worker-3@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com.celery.pidbox Y 13 13 0 8.30k 8.30k 0 1 2 > resource_manager Y 0 14 14 0 10.8k 10.8k 1 2 > resource_manager@ec2-54-73-155-135.eu-west-1.compute.amazonaws.com.celery.pidbox Y 13 13 0 8.30k 8.30k 0 1 2 > >### assert the test cases got unblocked >[root@ec2-54-73-155-135 pulp_auto]# ls -ltrh /usr/share/pulp_auto/*.log | tail -1 >-rw-r--r--. 1 root root 751K May 15 08:59 /usr/share/pulp_auto/xunit-_r-1_w-1_b-100000_t-0.4.xml.log >[root@ec2-54-73-155-135 pulp_auto]# ps -ef | grep [t]est >root 21025 26502 4 09:01 ? 00:00:15 /bin/python /usr/bin/nosetests -v --with-xunit --xunit-file=xunit-_r-1_w-1_b-200000_t-1.xml >root 26502 1 0 May13 ? 00:00:00 /bin/bash test.sh >[root@ec2-54-73-155-135 pulp_auto]# >
<b>You cannot view the attachment while viewing its details because your browser does not support IFRAMEs. <a href="attachment.cgi?id=895814">View the attachment on a separate page</a>.</b>
View Attachment As Raw
Actions:
View
Attachments on
bug 1088481
:
886965
|
895511
| 895814