Attachment 1450871 Details for Bug 1590779 – foreman-tail output during the issue

foreman-tail output during the issue

foreman-tail (text/plain), 50.07 KB, created by hprakash on 2018-06-13 11:56:00 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: hprakash

Created: 2018-06-13 11:56:00 UTC

Size: 50.07 KB

patch

obsolete

>==> /var/log/messages <==
>Jun  8 21:28:19 vm253-191 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.74.253.191 - - [08/Jun/2018:21:28:19 +0530] "GET /pulp/api/v2//status/ HTTP/1.1" 200 260 "-" "Ruby"
>10.74.253.191 - admin [08/Jun/2018:21:28:19 +0530] "GET /pulp/api/v2/users/ HTTP/1.1" 200 139 "-" "Ruby"
> 
>==> /var/log/httpd/foreman-ssl_error_ssl.log <==
>[Fri Jun 08 21:28:19.356613 2018] [ssl:warn] [pid 2021] [client 10.74.253.191:36030] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:19 72ec348c [app] [I] Started POST "/katello/api/v2/repositories/246/sync?organization_id=1" for 10.76.0.93 at 2018-06-08 21:28:19 +0530
>2018-06-08 21:28:19 72ec348c [app] [I] Processing by Katello::Api::V2::RepositoriesController#sync as JSON
>2018-06-08 21:28:19 72ec348c [app] [I]   Parameters: {"id"=>"246", "organization_id"=>"1", "api_version"=>"v2", "repository"=>{"id"=>"246"}}
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:19 +0530] "POST /katello/api/v2/repositories/246/sync?organization_id=1 HTTP/1.1" 202 1283 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/repositories/246" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
>10.76.0.93 - - [08/Jun/2018:21:28:19 +0530] "GET /tasks/views/task-details.html HTTP/1.1" 304 - "https://vm253-191.gsslab.pnq2.redhat.com/products/186/repositories/246" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
>10.76.0.93 - - [08/Jun/2018:21:28:19 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 660 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/repositories/246" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:19 72ec348c [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.170/app/views/katello/api/v2/repositories/sync.json.rabl within katello/api/v2/layouts/resource (87.4ms)
>2018-06-08 21:28:19 72ec348c [app] [I] Completed 202 Accepted in 654ms (Views: 127.4ms | ActiveRecord: 52.0ms)
>2018-06-08 21:28:19 4e90587c [app] [I] Started GET "/tasks/views/task-details.html" for 10.76.0.93 at 2018-06-08 21:28:19 +0530
>2018-06-08 21:28:19 897840b5 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:19 +0530
>2018-06-08 21:28:19 897840b5 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:19 897840b5 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:19 897840b5 [app] [I] Completed 200 OK in 55ms (Views: 1.1ms | ActiveRecord: 3.2ms)
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:21 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 660 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:21 07faaa1c [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:21 +0530
>2018-06-08 21:28:21 07faaa1c [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:21 07faaa1c [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:21 07faaa1c [app] [I] Completed 200 OK in 43ms (Views: 1.1ms | ActiveRecord: 3.2ms)
> 
>==> /var/log/messages <==
>Jun  8 21:28:23 vm253-191 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[b32b7e65-f4ca-4514-8c9d-aa371e060329]
>Jun  8 21:28:23 vm253-191 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[fac3b034-5e5b-4974-a8ad-15ce0ac475cd]
>Jun  8 21:28:23 vm253-191 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[b32b7e65-f4ca-4514-8c9d-aa371e060329] succeeded in 0.0431799939834s: None
>Jun  8 21:28:23 vm253-191 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[cf5d889c-53e7-4942-aae8-2fda0ddc2d6f]
>Jun  8 21:28:23 vm253-191 pulp: py.warnings:WARNING: (31718-58336) /usr/lib/python2.7/site-packages/mongoengine/queryset/base.py:461: DeprecationWarning: update is deprecated. Use replace_one, update_one or update_many instead.
>Jun  8 21:28:23 vm253-191 pulp: py.warnings:WARNING: (31718-58336)   upsert=upsert, **write_concern)
>Jun  8 21:28:23 vm253-191 pulp: py.warnings:WARNING: (31718-58336)
>Jun  8 21:28:23 vm253-191 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): registry.access.redhat.com
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.74.253.191 - admin [08/Jun/2018:21:28:23 +0530] "POST /pulp/api/v2/repositories/redhat-docker-openshift3/actions/sync/ HTTP/1.1" 202 172 "-" "Ruby"
>10.74.253.191 - admin [08/Jun/2018:21:28:23 +0530] "GET /pulp/api/v2/tasks/fac3b034-5e5b-4974-a8ad-15ce0ac475cd/ HTTP/1.1" 200 322 "-" "Ruby"
>10.76.0.93 - - [08/Jun/2018:21:28:23 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 659 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:23 5d72bce1 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:23 +0530
>2018-06-08 21:28:23 5d72bce1 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:23 5d72bce1 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:23 5d72bce1 [app] [I] Completed 200 OK in 41ms (Views: 2.0ms | ActiveRecord: 4.2ms)
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.74.253.191 - admin [08/Jun/2018:21:28:23 +0530] "GET /pulp/api/v2/tasks/fac3b034-5e5b-4974-a8ad-15ce0ac475cd/ HTTP/1.1" 200 379 "-" "Ruby"
> 
>==> /var/log/messages <==
>Jun  8 21:28:24 vm253-191 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: registry.access.redhat.com
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:24 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 670 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
>10.74.253.191 - admin [08/Jun/2018:21:28:24 +0530] "GET /pulp/api/v2/tasks/fac3b034-5e5b-4974-a8ad-15ce0ac475cd/ HTTP/1.1" 200 775 "-" "Ruby"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:24 108fa4a9 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:24 +0530
>2018-06-08 21:28:24 108fa4a9 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:24 108fa4a9 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:24 108fa4a9 [app] [I] Completed 200 OK in 41ms (Views: 1.0ms | ActiveRecord: 2.9ms)
> 
>==> /var/log/messages <==
>Jun  8 21:28:25 vm253-191 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): access.redhat.com
>Jun  8 21:28:25 vm253-191 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): registry.redhat.io
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.74.253.191 - admin [08/Jun/2018:21:28:25 +0530] "GET /pulp/api/v2/tasks/fac3b034-5e5b-4974-a8ad-15ce0ac475cd/ HTTP/1.1" 200 775 "-" "Ruby"
>10.76.0.93 - - [08/Jun/2018:21:28:26 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 670 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:26 940e4976 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:26 +0530
>2018-06-08 21:28:26 940e4976 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:26 940e4976 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:26 940e4976 [app] [I] Completed 200 OK in 54ms (Views: 1.1ms | ActiveRecord: 3.8ms)
> 
>==> /var/log/messages <==
>Jun  8 21:28:26 vm253-191 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: registry.access.redhat.com
>Jun  8 21:28:28 vm253-191 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: registry.access.redhat.com
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.74.253.191 - admin [08/Jun/2018:21:28:27 +0530] "GET /pulp/api/v2/tasks/fac3b034-5e5b-4974-a8ad-15ce0ac475cd/ HTTP/1.1" 200 775 "-" "Ruby"
>10.76.0.93 - - [08/Jun/2018:21:28:27 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 670 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:27 cf25a819 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:27 +0530
>2018-06-08 21:28:27 cf25a819 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:27 cf25a819 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:27 cf25a819 [app] [I] Completed 200 OK in 46ms (Views: 1.6ms | ActiveRecord: 3.1ms)
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:29 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 670 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:29 aad35888 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:29 +0530
>2018-06-08 21:28:29 aad35888 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:29 aad35888 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:29 aad35888 [app] [I] Completed 200 OK in 55ms (Views: 2.2ms | ActiveRecord: 4.1ms)
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.74.253.191 - admin [08/Jun/2018:21:28:29 +0530] "GET /pulp/api/v2/tasks/fac3b034-5e5b-4974-a8ad-15ce0ac475cd/ HTTP/1.1" 200 775 "-" "Ruby"
> 
>==> /var/log/messages <==
>Jun  8 21:28:30 vm253-191 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: registry.access.redhat.com
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:30 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 670 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
>10.74.253.191 - admin [08/Jun/2018:21:28:31 +0530] "GET /pulp/api/v2/tasks/fac3b034-5e5b-4974-a8ad-15ce0ac475cd/ HTTP/1.1" 200 775 "-" "Ruby"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:30 00b0d9d4 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:30 +0530
>2018-06-08 21:28:30 00b0d9d4 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:30 00b0d9d4 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:30 00b0d9d4 [app] [I] Completed 200 OK in 43ms (Views: 1.2ms | ActiveRecord: 3.5ms)
> 
>==> /var/log/messages <==
>Jun  8 21:28:32 vm253-191 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: registry.access.redhat.com
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:32 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 670 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
>10.74.253.191 - admin [08/Jun/2018:21:28:33 +0530] "GET /pulp/api/v2/tasks/fac3b034-5e5b-4974-a8ad-15ce0ac475cd/ HTTP/1.1" 200 775 "-" "Ruby"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:32 1a437311 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:32 +0530
>2018-06-08 21:28:32 1a437311 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:32 1a437311 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:32 1a437311 [app] [I] Completed 200 OK in 40ms (Views: 1.3ms | ActiveRecord: 3.0ms)
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:34 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 670 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:34 f3c70432 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:34 +0530
>2018-06-08 21:28:34 f3c70432 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:34 f3c70432 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:34 f3c70432 [app] [I] Completed 200 OK in 46ms (Views: 1.8ms | ActiveRecord: 2.8ms)
> 
>==> /var/log/messages <==
>Jun  8 21:28:34 vm253-191 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: registry.access.redhat.com
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.74.253.191 - admin [08/Jun/2018:21:28:35 +0530] "GET /pulp/api/v2/tasks/fac3b034-5e5b-4974-a8ad-15ce0ac475cd/ HTTP/1.1" 200 775 "-" "Ruby"
> 
>==> /var/log/messages <==
>Jun  8 21:28:36 vm253-191 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: registry.access.redhat.com
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:35 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 670 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:35 207a061d [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:35 +0530
>2018-06-08 21:28:35 207a061d [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:35 207a061d [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:35 207a061d [app] [I] Completed 200 OK in 40ms (Views: 1.1ms | ActiveRecord: 2.7ms)
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.74.253.191 - admin [08/Jun/2018:21:28:37 +0530] "GET /pulp/api/v2/tasks/fac3b034-5e5b-4974-a8ad-15ce0ac475cd/ HTTP/1.1" 200 775 "-" "Ruby"
>10.76.0.93 - - [08/Jun/2018:21:28:37 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 670 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:37 7826db14 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:37 +0530
>2018-06-08 21:28:37 7826db14 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:37 7826db14 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:37 7826db14 [app] [I] Completed 200 OK in 41ms (Views: 1.6ms | ActiveRecord: 3.7ms)
> 
>==> /var/log/messages <==
>Jun  8 21:28:38 vm253-191 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: registry.access.redhat.com
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:38 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 670 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
>10.74.253.191 - admin [08/Jun/2018:21:28:39 +0530] "GET /pulp/api/v2/tasks/fac3b034-5e5b-4974-a8ad-15ce0ac475cd/ HTTP/1.1" 200 775 "-" "Ruby"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:38 1b0a4cca [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:38 +0530
>2018-06-08 21:28:38 1b0a4cca [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:38 1b0a4cca [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:38 1b0a4cca [app] [I] Completed 200 OK in 38ms (Views: 1.1ms | ActiveRecord: 3.2ms)
> 
>==> /var/log/messages <==
>Jun  8 21:28:39 vm253-191 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: registry.access.redhat.com
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:40 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 670 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:40 29dc5530 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:40 +0530
>2018-06-08 21:28:40 29dc5530 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:40 29dc5530 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:40 29dc5530 [app] [I] Completed 200 OK in 46ms (Views: 1.3ms | ActiveRecord: 5.2ms)
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.74.253.191 - admin [08/Jun/2018:21:28:41 +0530] "GET /pulp/api/v2/tasks/fac3b034-5e5b-4974-a8ad-15ce0ac475cd/ HTTP/1.1" 200 775 "-" "Ruby"
> 
>==> /var/log/messages <==
>Jun  8 21:28:41 vm253-191 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: registry.access.redhat.com
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:41 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 670 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:41 fd7e9021 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:41 +0530
>2018-06-08 21:28:41 fd7e9021 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:41 fd7e9021 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:41 fd7e9021 [app] [I] Completed 200 OK in 34ms (Views: 1.0ms | ActiveRecord: 2.6ms)
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.74.253.191 - admin [08/Jun/2018:21:28:43 +0530] "GET /pulp/api/v2/tasks/fac3b034-5e5b-4974-a8ad-15ce0ac475cd/ HTTP/1.1" 200 775 "-" "Ruby"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:43 6f45a660 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:43 +0530
>2018-06-08 21:28:43 6f45a660 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:43 6f45a660 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
> 
>==> /var/log/messages <==
>Jun  8 21:28:43 vm253-191 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: registry.access.redhat.com
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:43 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 670 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:43 6f45a660 [app] [I] Completed 200 OK in 48ms (Views: 2.1ms | ActiveRecord: 3.6ms)
> 
>==> /var/log/messages <==
>Jun  8 21:28:45 vm253-191 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: registry.access.redhat.com
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:45 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 670 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
>10.74.253.191 - admin [08/Jun/2018:21:28:45 +0530] "GET /pulp/api/v2/tasks/fac3b034-5e5b-4974-a8ad-15ce0ac475cd/ HTTP/1.1" 200 775 "-" "Ruby"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:45 06362f00 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:45 +0530
>2018-06-08 21:28:45 06362f00 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:45 06362f00 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:45 06362f00 [app] [I] Completed 200 OK in 35ms (Views: 0.9ms | ActiveRecord: 2.6ms)
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:46 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 670 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
>10.74.253.191 - admin [08/Jun/2018:21:28:47 +0530] "GET /pulp/api/v2/tasks/fac3b034-5e5b-4974-a8ad-15ce0ac475cd/ HTTP/1.1" 200 775 "-" "Ruby"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:46 8984da03 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:46 +0530
>2018-06-08 21:28:46 8984da03 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:46 8984da03 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:46 8984da03 [app] [I] Completed 200 OK in 39ms (Views: 1.0ms | ActiveRecord: 3.6ms)
> 
>==> /var/log/messages <==
>Jun  8 21:28:48 vm253-191 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: registry.access.redhat.com
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:48 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 670 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:48 289c4cb7 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:48 +0530
>2018-06-08 21:28:48 289c4cb7 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:48 289c4cb7 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:48 289c4cb7 [app] [I] Completed 200 OK in 43ms (Views: 1.5ms | ActiveRecord: 3.1ms)
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.74.253.191 - admin [08/Jun/2018:21:28:49 +0530] "GET /pulp/api/v2/tasks/fac3b034-5e5b-4974-a8ad-15ce0ac475cd/ HTTP/1.1" 200 775 "-" "Ruby"
> 
>==> /var/log/messages <==
>Jun  8 21:28:50 vm253-191 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: registry.access.redhat.com
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:49 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 670 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:49 7388ff97 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:49 +0530
>2018-06-08 21:28:49 7388ff97 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:49 7388ff97 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:49 7388ff97 [app] [I] Completed 200 OK in 51ms (Views: 1.0ms | ActiveRecord: 2.5ms)
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:51 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 670 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:51 e8da86be [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:51 +0530
>2018-06-08 21:28:51 e8da86be [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:51 e8da86be [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:51 e8da86be [app] [I] Completed 200 OK in 45ms (Views: 1.5ms | ActiveRecord: 3.5ms)
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:52 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 670 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
>10.74.253.191 - admin [08/Jun/2018:21:28:53 +0530] "GET /pulp/api/v2/tasks/fac3b034-5e5b-4974-a8ad-15ce0ac475cd/ HTTP/1.1" 200 775 "-" "Ruby"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:52 3180295e [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:52 +0530
>2018-06-08 21:28:52 3180295e [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:52 3180295e [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:52 3180295e [app] [I] Completed 200 OK in 40ms (Views: 1.0ms | ActiveRecord: 2.8ms)
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:54 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 670 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:54 c241434b [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:54 +0530
>2018-06-08 21:28:54 c241434b [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:54 c241434b [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:54 c241434b [app] [I] Completed 200 OK in 44ms (Views: 1.0ms | ActiveRecord: 3.1ms)
> 
>==> /var/log/messages <==
>Jun  8 21:28:55 vm253-191 pulp: py.warnings:WARNING: (31718-58336) /usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py:806: DeprecationWarning: save is deprecated. Use insert_one or replace_one instead
>Jun  8 21:28:55 vm253-191 pulp: py.warnings:WARNING: (31718-58336)   sync_result_collection.save(sync_result)
>Jun  8 21:28:55 vm253-191 pulp: py.warnings:WARNING: (31718-58336)
>Jun  8 21:28:55 vm253-191 pulp: pulp.server.async.tasks:INFO: Task failed : [fac3b034-5e5b-4974-a8ad-15ce0ac475cd]
>Jun  8 21:28:55 vm253-191 pulp: py.warnings:WARNING: (31718-58336) /usr/lib/python2.7/site-packages/mongoengine/document.py:367: DeprecationWarning: update is deprecated. Use replace_one, update_one or update_many instead.
>Jun  8 21:28:55 vm253-191 pulp: py.warnings:WARNING: (31718-58336)   upsert=upsert, **write_concern)
>Jun  8 21:28:55 vm253-191 pulp: py.warnings:WARNING: (31718-58336)
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336) Task pulp.server.managers.repo.sync.sync[fac3b034-5e5b-4974-a8ad-15ce0ac475cd] raised unexpected: IOError('Not Found',)
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336) Traceback (most recent call last):
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)     R = retval = fun(*args, **kwargs)
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 478, in __call__
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)     return super(Task, self).__call__(*args, **kwargs)
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)     return super(PulpTask, self).__call__(*args, **kwargs)
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)     return self.run(*args, **kwargs)
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 762, in sync
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)     sync_report = sync_repo(transfer_repo, conduit, call_config)
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 668, in wrap_f
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)     return f(*args, **kwargs)
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)   File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 84, in sync_repo
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)     return self.sync_step.process_lifecycle()
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 562, in process_lifecycle
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)     super(PluginStep, self).process_lifecycle()
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 159, in process_lifecycle
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)     step.process()
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 249, in process
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)     self._process_block()
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 293, in _process_block
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)     self.process_main()
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)   File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/sync.py", line 229, in process_main
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)     digest, manifest = self.parent.index_repository.get_manifest(tag)
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)   File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/registry.py", line 366, in get_manifest
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)     headers, manifest = self._get_path(path)
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)   File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/registry.py", line 431, in _get_path
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)     self._raise_path_error(report)
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)   File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/registry.py", line 453, in _raise_path_error
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336)     raise IOError(report.error_msg)
>Jun  8 21:28:55 vm253-191 pulp: celery.worker.job:ERROR: (1204-58336) IOError: Not Found
>Jun  8 21:28:55 vm253-191 pulp: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
>Jun  8 21:28:55 vm253-191 pulp: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
>Jun  8 21:28:56 vm253-191 pulp: pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
>Jun  8 21:28:56 vm253-191 pulp: py.warnings:WARNING: (31850-58336) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing>
>Jun  8 21:28:56 vm253-191 pulp: py.warnings:WARNING: (31850-58336)   "MongoClient opened before fork. Create MongoClient "
>Jun  8 21:28:56 vm253-191 pulp: py.warnings:WARNING: (31850-58336)
>Jun  8 21:28:56 vm253-191 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[cf5d889c-53e7-4942-aae8-2fda0ddc2d6f] succeeded in 0.012702447013s: None
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:55 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 670 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:55 c54dc2f7 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:55 +0530
>2018-06-08 21:28:56 c54dc2f7 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:56 c54dc2f7 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:56 c54dc2f7 [app] [I] Completed 200 OK in 49ms (Views: 1.4ms | ActiveRecord: 6.7ms)
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.74.253.191 - admin [08/Jun/2018:21:28:57 +0530] "GET /pulp/api/v2/tasks/fac3b034-5e5b-4974-a8ad-15ce0ac475cd/ HTTP/1.1" 200 1384 "-" "Ruby"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:57  [foreman-tasks/action] [E] PLP0000: Not Found (Katello::Errors::PulpError)
> | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.170/app/lib/actions/pulp/abstract_async_task.rb:121:in `block in external_task='
> | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.170/app/lib/actions/pulp/abstract_async_task.rb:119:in `each'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.170/app/lib/actions/pulp/abstract_async_task.rb:119:in `external_task='
> | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.170/app/lib/actions/pulp/repository/sync.rb:48:in `external_task='
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/action/polling.rb:98:in `poll_external_task_with_rescue'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/action/polling.rb:21:in `run'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/action/cancellable.rb:9:in `run'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.170/app/lib/actions/pulp/abstract_async_task.rb:45:in `run'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/action.rb:506:in `block (3 levels) in execute_run'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/middleware/stack.rb:26:in `call'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/middleware/stack.rb:26:in `pass'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/middleware.rb:17:in `pass'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/middleware.rb:30:in `run'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/middleware/stack.rb:22:in `call'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/middleware/stack.rb:26:in `pass'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/middleware.rb:17:in `pass'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.170/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.170/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.170/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.170/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.170/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.170/app/lib/actions/middleware/remote_action.rb:16:in `run'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/middleware/stack.rb:22:in `call'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/middleware/stack.rb:26:in `pass'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/middleware.rb:17:in `pass'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/action/progress.rb:16:in `run'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/middleware/stack.rb:22:in `call'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/middleware/stack.rb:26:in `pass'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/middleware.rb:17:in `pass'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.170/app/lib/actions/middleware/keep_locale.rb:11:in `block in run'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.170/app/lib/actions/middleware/keep_locale.rb:22:in `with_locale'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.170/app/lib/actions/middleware/keep_locale.rb:11:in `run'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/middleware/stack.rb:22:in `call'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/middleware/stack.rb:26:in `pass'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/middleware.rb:17:in `pass'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/middleware.rb:30:in `run'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/middleware/stack.rb:22:in `call'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/middleware/world.rb:30:in `execute'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/action.rb:505:in `block (2 levels) in execute_run'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/action.rb:504:in `catch'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/action.rb:504:in `block in execute_run'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/action.rb:419:in `call'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/action.rb:419:in `block in with_error_handling'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/action.rb:419:in `catch'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/action.rb:419:in `with_error_handling'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/action.rb:499:in `execute_run'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/action.rb:260:in `execute'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/executors/parallel/worker.rb:12:in `on_message'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.6/lib/dynflow/actor.rb:26:in `on_envelope'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in `process_envelope'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:in `block in on_envelope'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:in `work'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `call'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `run_task'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in `block (3 levels) in create_worker'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `loop'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `block (2 levels) in create_worker'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `catch'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `block in create_worker'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'
> | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:57 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 680 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:57 3855a90c [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:57 +0530
>2018-06-08 21:28:57 3855a90c [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:57 3855a90c [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:57 3855a90c [app] [I] Completed 200 OK in 38ms (Views: 1.4ms | ActiveRecord: 3.2ms)
> 
>==> /var/log/httpd/foreman-ssl_access_ssl.log <==
>10.76.0.93 - - [08/Jun/2018:21:28:59 +0530] "POST /foreman_tasks/api/tasks/bulk_search HTTP/1.1" 200 684 "https://vm253-191.gsslab.pnq2.redhat.com/products/186/tasks/47769dcd-0fec-41b3-a49e-0be8d08baea6" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
> 
>==> /var/log/foreman/production.log <==
>2018-06-08 21:28:59 6d72d145 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.76.0.93 at 2018-06-08 21:28:59 +0530
>2018-06-08 21:28:59 6d72d145 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
>2018-06-08 21:28:59 6d72d145 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"47769dcd-0fec-41b3-a49e-0be8d08baea6", "search_id"=>"1"}], "task"=>{}}
>2018-06-08 21:28:59 6d72d145 [app] [I] Completed 200 OK in 58ms (Views: 1.3ms | ActiveRecord: 13.2ms)

Actions: View

Attachments on bug 1590779: 1450871