Bug 1323644

Summary: Capsule sync failed while syncing ostree contents
Product: Red Hat Satellite Reporter: Sachin Ghai <sghai>
Component: Foreman ProxyAssignee: John Mitsch <jomitsch>
Status: CLOSED ERRATA QA Contact: Sachin Ghai <sghai>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2.0CC: bbuckingham, ehelms, jomitsch, ktordeur, mhrivnak, mmccune, sghai
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
URL: http://projects.theforeman.org/issues/15745
Whiteboard:
Fixed In Version: libtasn1-3.8-3,rubygem-katello-3.0.0.66-1,pulp-ostree-1.1.1-2 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-27 11:13:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1330753, 1382094    
Bug Blocks:    
Attachments:
Description Flags
uninitialized constant Actions::Pulp::AbstractAsyncTask::PulpError in logs
none
pulp exceptions on capsule node, while running sync between sat server and capsule none

Description Sachin Ghai 2016-04-04 10:30:40 UTC 
Description of problem: I was trying to sync contents from Satellite server to external capsule and sync failed as below:


~]# hammer -u admin -p changeme capsule content synchronize --id 2
[......................................................................                                                                       ] [50%]
400 Bad Request
400 Bad Request
400 Bad Request
400 Bad Request
400 Bad Request
400 Bad Request
404 Resource Not Found
404 Resource Not Found
404 Resource Not Found
404 Resource Not Found
404 Resource Not Found
404 Resource Not Found


Later, I checked the /var/log/messages on external capsule and found following exception:

Apr  4 06:16:35 cloud-qe-7 pulp: pulp.server.webservices.middleware.exception:INFO: Missing resource(s): repository=Default_Organization-Dev-cv_ostree-Red_Hat_Enterprise_Linux_Atomic_Host-Red_Hat_Enterprise_Linux_Atomic_Host_Trees
Apr  4 06:16:36 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-88416) Exception adding importer to repo [Default_Organization-Dev-custom_ostree-ostree-f23_ostree]; the repo will be deleted
Apr  4 06:16:36 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-88416) Traceback (most recent call last):
Apr  4 06:16:36 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-88416)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 410, in create_repo
Apr  4 06:16:36 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-88416)     importer_controller.set_importer(repo_id, importer_type_id, importer_repo_plugin_config)
Apr  4 06:16:36 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-88416)   File "/usr/lib/python2.7/site-packages/celery/local.py", line 167, in <lambda>
Apr  4 06:16:36 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-88416)     __call__ = lambda x, *a, **kw: x._get_current_object()(*a, **kw)
Apr  4 06:16:36 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-88416)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__
Apr  4 06:16:36 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-88416)     return super(Task, self).__call__(*args, **kwargs)
Apr  4 06:16:36 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-88416)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Apr  4 06:16:36 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-88416)     return super(PulpTask, self).__call__(*args, **kwargs)
Apr  4 06:16:36 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-88416)   File "/usr/lib/python2.7/site-packages/celery/app/task.py", line 420, in __call__
Apr  4 06:16:36 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-88416)     return self.run(*args, **kwargs)
Apr  4 06:16:36 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-88416)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/importer.py", line 72, in set_importer
Apr  4 06:16:36 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-88416)     validate_importer_config(repo.repo_id, importer_type_id, repo_plugin_config)
Apr  4 06:16:36 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-88416)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/importer.py", line 144, in validate_importer_config
Apr  4 06:16:36 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-88416)     importer_type_id=importer_type_id)
Apr  4 06:16:36 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-88416) PulpCodedValidationException: The importer type ostree_web_importer does not exist


Looks like sync failed while syncing ostree contents to capsule  and that's because of missing installed plugin: pulp-ostree-plugins

I installed pulp-ostree-plugin on capsule and later ran installer as below:


[root@cloud-qe-7 ~]# foreman-installer --scenario capsule --enable-ostree true
Installing             Done                                               [100%] [..................................................................]
  Success!
  The full log is at /var/log/foreman-installer/capsule.log


But still I'm getting:

Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31427-95712) PulpCodedValidationException: The importer type ostree_web_importer does not exist
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.webservices.middleware.exception:INFO: The importer type ostree_web_importer does not exist
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31427-92576) Exception adding distributor to repo [Default_Organization-Dev-cv_rhel7-RHEL7-RHEL7_x86_64]; the repo will be deleted
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31427-92576) Traceback (most recent call last):
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31427-92576)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 425, in create_repo
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31427-92576)     dist_controller.add_distributor(repo_id, type_id, plugin_config, auto_publish, dist_id)
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31427-92576)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/distributor.py", line 50, in add_distributor
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31427-92576)     raise exceptions.InvalidValue(['distributor_type_id'])
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31427-92576) InvalidValue: Invalid properties: ['distributor_type_id']
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.webservices.middleware.exception:INFO: Invalid properties: ['distributor_type_id']
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-81120) Exception adding importer to repo [Default_Organization-Dev-custom_ostree-ostree-f23_ostree]; the repo will be deleted
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-81120) Traceback (most recent call last):
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-81120)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 410, in create_repo
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-81120)     importer_controller.set_importer(repo_id, importer_type_id, importer_repo_plugin_config)
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-81120)   File "/usr/lib/python2.7/site-packages/celery/local.py", line 167, in <lambda>
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-81120)     __call__ = lambda x, *a, **kw: x._get_current_object()(*a, **kw)
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-81120)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-81120)     return super(Task, self).__call__(*args, **kwargs)
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-81120)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-81120)     return super(PulpTask, self).__call__(*args, **kwargs)
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-81120)   File "/usr/lib/python2.7/site-packages/celery/app/task.py", line 420, in __call__
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-81120)     return self.run(*args, **kwargs)
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-81120)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/importer.py", line 72, in set_importer
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-81120)     validate_importer_config(repo.repo_id, importer_type_id, repo_plugin_config)
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-81120)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/importer.py", line 144, in validate_importer_config
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-81120)     importer_type_id=importer_type_id)
Apr  4 06:26:51 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-81120) PulpCodedValidationException: The importer type ostree_web_importer does not exist


Version-Release number of selected component (if applicable):
Satellite 6.2 beta snap6

How reproducible:
always

Steps to Reproduce:
1. Install satellite server, sync ostree contents
2. promote/publish them to dev env
3. sync contents to external capsule

Actual results:

sync failed:

(31428-81120) PulpCodedValidationException: The importer type ostree_web_importer does not exist



Expected results:
sync should succeed.

Additional info:
Comment 2 Sachin Ghai 2016-04-04 11:25:45 UTC 
root@cloud-qe-7 ~]# tail -f /var/log/messages 
Apr  4 07:01:08 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-07168)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/distributor.py", line 50, in add_distributor
Apr  4 07:01:08 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-07168)     raise exceptions.InvalidValue(['distributor_type_id'])
Apr  4 07:01:08 cloud-qe-7 pulp: pulp.server.controllers.repository:ERROR: (31428-07168) InvalidValue: Invalid properties: ['distributor_type_id']
Apr  4 07:01:08 cloud-qe-7 pulp: pulp.server.webservices.middleware.exception:INFO: Invalid properties: ['distributor_type_id']
Comment 3 Michael Hrivnak 2016-04-04 12:36:50 UTC 
After installing a new plugin, you have to run pulp-manage-db as the "apache" user, and then restart all pulp processes. Hopefully those steps will get your system working.
Comment 4 Sachin Ghai 2016-04-05 05:47:01 UTC 
Thanks Michael for reply.

I ran "pulp-manage-db" as "apache" user and restarted all pulp services but this time, I got new error in addition to earlier ones.


 ~]# sudo -u apache pulp-manage-db
Attempting to connect to localhost:27017
Attempting to connect to localhost:27017
Write concern for Mongo connection: {}
Loading content types.
Loading type descriptors []
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: ostree = pulp_ostree.plugins.db.model:Branch
Updating the database with types []
Found the following type definitions that were not present in the update collection [puppet_module, docker_tag, docker_manifest, docker_blob, erratum, yum_repo_metadata_file, package_group, package_category, iso, package_environment, drpm, srpm, rpm, distribution, docker_image]
Updating the database with types [puppet_module, docker_tag, ostree, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, drpm, distribution, rpm, srpm, docker_image]
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
Migration package pulp.server.db.migrations is up to date at version 22
Migration package pulp_docker.plugins.migrations is up to date at version 1
Migration package pulp_puppet.plugins.migrations is up to date at version 3
Migration package pulp_rpm.plugins.migrations is up to date at version 27
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: ostree = pulp_ostree.plugins.db.model:Branch
Database migrations complete.


~]# for i in qpidd pulp_workers pulp_celerybeat pulp_resource_manager httpd; do service $i restart; done
Redirecting to /bin/systemctl restart  qpidd.service
Redirecting to /bin/systemctl restart  pulp_workers.service
Redirecting to /bin/systemctl restart  pulp_celerybeat.service
Redirecting to /bin/systemctl restart  pulp_resource_manager.service
Redirecting to /bin/systemctl restart  httpd.service



[root@cloud-qe ~]# hammer -u admin -p changeme capsule content synchronize --id 2
[..............................................................................................                                               ] [66%]
400 Bad Request
400 Bad Request
400 Bad Request
404 Resource Not Found
404 Resource Not Found
404 Resource Not Found
uninitialized constant Actions::Pulp::AbstractAsyncTask::PulpError
uninitialized constant Actions::Pulp::AbstractAsyncTask::PulpError
uninitialized constant Actions::Pulp::AbstractAsyncTask::PulpError
Comment 6 Michael Hrivnak 2016-04-06 01:27:29 UTC 
Can you look in the system log for anything that pulp logged while that sync was running, and especially any errors?
Comment 7 Sachin Ghai 2016-04-06 11:02:27 UTC 
Created attachment 1144190 [details]
uninitialized constant Actions::Pulp::AbstractAsyncTask::PulpError in logs
Comment 8 Sachin Ghai 2016-04-06 11:04:00 UTC 
Yes, I can see following errors, complete logs are in attachement in comment7

2016-04-06 06:58:49 [foreman-tasks/action] [E] uninitialized constant Actions::Pulp::AbstractAsyncTask::PulpError (NameError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_hooks-0.3.9/lib/foreman_hooks.rb:83:in `load_missing_constant_with_hooks'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.12/app/lib/actions/pulp/abstract_async_task.rb:95:in `rescue_external_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/polling.rb:102:in `rescue in poll_external_task_with_rescue'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/polling.rb:97:in `poll_external_task_with_rescue'



2016-04-06 06:58:52 [app] [D] Body: {"id":"5c5d7ca6-8619-4421-b1f1-acc519588d4b","label":"Actions::Katello::CapsuleContent::Sync","pending":true,"username":"admin","started_at":"2016-04-06 10:58:45 UTC","ended_at":null,"state":"paused","result":"error","progress":0.7692307692307693,"input":{"smart_proxy":{"id":2,"name":"cloud-qe-7.idmqe.lab.eng.bos.redhat.com"},"services_checked":["pulp","pulp_auth"],"locale":"en"},"output":{},"humanized":{"action":"Synchronize capsule content","input":[],"output":"","errors":["400 Bad Request","400 Bad Request","400 Bad Request","404 Resource Not Found","404 Resource Not Found","404 Resource Not Found","uninitialized constant Actions::Pulp::AbstractAsyncTask::PulpError","uninitialized constant Actions::Pulp::AbstractAsyncTask::PulpError","uninitialized constant Actions::Pulp::AbstractAsyncTask::PulpError"]},"cli_example":null}
2016-04-06 06:58:52 [app] [I] Completed 200 OK in 64ms (Views: 33.6ms | ActiveRecord: 20.7ms)
Comment 9 Michael Hrivnak 2016-04-06 18:30:19 UTC 
The attachment just has katello output. I'm not sure what the errors mean that it's showing. Brad, can you suggest next steps?

Sachin, somewhere on your system, pulp logs output. Upstream it all goes to the system log. I'm not sure if Satellite has it configured to log elsewhere. If you could find log statements from pulp, that would be helpful.
Comment 10 Sachin Ghai 2016-04-07 07:10:24 UTC 
@Michael: Pulp logs generated under '/var/log/messages' afaik. However, I don't see any log on satellite server while running capsule sync. However, I can see some pulp exception on capsule node while running sync. Please see the attachment.
Comment 11 Sachin Ghai 2016-04-07 07:11:05 UTC 
Created attachment 1144568 [details]
pulp exceptions on capsule node, while running sync between sat server and capsule
Comment 12 Sachin Ghai 2016-04-13 11:04:15 UTC 
Okay, I installed fresh setup with sat6.2 beta snap7.1 and capsule sync. Sync was completed with 100% but two issues:


[root@cloud-qe-3 ~]# hammer -u admin -p changeme capsule content synchronize --id=2
[............................................................................................................................................] [100%]
OST0005: Fetch summary failed. Reason: GLib.Error('Server required TLS certificate', 'g-tls-error-quark', 5)
DKR1008: Could not find registry API at https://cloud-qe-3.idmqe.lab.eng.bos.redhat.com:5000


I've synced ostree contents and docker contents on sat server. Please note that ostree plugin is enabled on capsule node.

 ~]# foreman-installer --scenario capsule --enable-ostree true
Installing             Done                                               [100%] [..................................................................]
  Success!
Comment 13 Sachin Ghai 2016-04-13 11:07:33 UTC 
here is exception for error:

DKR1008: Could not find registry API at https://cloud-qe-3.idmqe.lab.eng.bos.redhat.com:5000

Please note that I've synced centos image from docker hub. Published and promoted in a CV to dev environment.

2016-04-13 06:27:26 [foreman-tasks/action] [E] DKR1008: Could not find registry API at cloud-qe-3.idmqe.lab.eng.bos.redhat.com:5000/default_organization-dev-cv_docker_centos-docker-centos (Katello::Errors::PulpError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/pulp/abstract_async_task.rb:121:in `block in external_task='
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/pulp/abstract_async_task.rb:119:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/pulp/abstract_async_task.rb:119:in `external_task='
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/polling.rb:98:in `poll_external_task_with_rescue'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/polling.rb:21:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/cancellable.rb:9:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/pulp/abstract_async_task.rb:45:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:506:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:30:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/models/katello/concerns/user_extensions.rb:20:in `cp_config'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/middleware/remote_action.rb:16:in `run'
 | /opt
Comment 14 Sachin Ghai 2016-04-13 11:10:17 UTC 
Other error assuming ostree related: 

OST0005: Fetch summary failed. Reason: GLib.Error('Server required TLS certificate', 'g-tls-error-quark', 5)

2016-04-13 06:42:44 [foreman-tasks/action] [E] OST0005: Fetch summary failed. Reason: GLib.Error('Server required TLS certificate', 'g-tls-error-quark', 5) (Katello::Errors::PulpError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/pulp/abstract_async_task.rb:121:in `block in external_task='
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/pulp/abstract_async_task.rb:119:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/pulp/abstract_async_task.rb:119:in `external_task='
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/polling.rb:98:in `poll_external_task_with_rescue'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/polling.rb:21:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/cancellable.rb:9:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/pulp/abstract_async_task.rb:45:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:506:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:30:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/models/katello/concerns/user_extensions.rb:20:in `cp_config'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/middleware/remote_action.rb:16:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/progress.rb:16:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/middleware/keep_locale.rb:11:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/middleware/keep_locale.rb:22:in `with_locale'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/middleware/keep_locale.rb:11:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:30:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/world.rb:30:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:505:in `block (2 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:504:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:504:in `block in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:499:in `execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:260:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/worker.rb:12:in `on_message'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
Comment 15 Sachin Ghai 2016-04-13 11:16:41 UTC 
On re-running sync, following error raised:

OST0004: Feed URL not specified
 ~]# hammer -u admin -p changeme capsule content synchronize --id=2
[............................................................................................................................................] [100%]
OST0004: Feed URL not specified
DKR1008: Could not find registry API at https://cloud-qe-3.idmqe.lab.eng.bos.redhat.com:5000



2016-04-13 07:13:49 [foreman-tasks/action] [E] OST0004: Feed URL not specified (Katello::Errors::PulpError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/pulp/abstract_async_task.rb:121:in `block in external_task='
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/pulp/abstract_async_task.rb:119:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/pulp/abstract_async_task.rb:119:in `external_task='
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/polling.rb:98:in `poll_external_task_with_rescue'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/polling.rb:21:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/cancellable.rb:9:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/pulp/abstract_async_task.rb:45:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:506:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:30:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/models/katello/concerns/user_extensions.rb:20:in `cp_config'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.15/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'




Please note that following errors are consistent in subsequent runs:

OST0004: Feed URL not specified
DKR1008: Could not find registry API at https://cloud-qe-3.idmqe.lab.eng.bos.redhat.com:5000
Comment 17 John Mitsch 2016-04-14 19:01:21 UTC 
This was fixed by upgrading the glib2 package on the capsule. I am currently looking into a permanent solution.
Comment 18 John Mitsch 2016-04-14 19:20:21 UTC 
The atomic team is being notified of this dependency issue. The gobject dependency on glib2 is not specific enough. Any further updates/trackers will be posted here.
Comment 19 John Mitsch 2016-04-15 20:10:28 UTC 
Just an update that the glib2 dependency issue is NOT related to this issue (but still something that the atomic team should resolve) sorry for any confusion.
Comment 20 Mike McCune 2016-04-19 21:05:48 UTC 
John:

Can you detail the exact workaround required to get around this issue?

What version of glib2 did you update to and where did you get it from?
Comment 21 John Mitsch 2016-04-19 21:12:57 UTC 
Mike,

This was all upstream, I had to upgrade to glib2-2.42.2-5.el7.x86_64

As I mentioned above, that was actually not the issue that was causing this bug, but rather a separate atomic issue that was similar (the atomic team knows about it now). We seem to have the same glib2 package downstream, so I think all is fine regarding that issue.

In short, disregard what I said about glib2 and I am continuing to investigate this bug.
Comment 22 Mike McCune 2016-04-20 02:26:19 UTC 
Apologies for the misread, thanks for the clarification!
Comment 24 John Mitsch 2016-04-27 17:24:01 UTC 
While a change does have to be made to our code for ostree repos to sync properly with capsules, the real blocker is a bug in atomic, which has been filed here https://bugzilla.redhat.com/show_bug.cgi?id=1330753
Comment 25 Mike McCune 2016-07-12 21:27:04 UTC 
This is awaiting the fix contained within: 

https://bugzilla.redhat.com/show_bug.cgi?id=1354608

marking POST as that bug is also in POST.
Comment 26 Sachin Ghai 2016-07-19 07:16:08 UTC 
Verified with sat62 GA snap20.2

Issue is reproducible with below error:

[root@cloud-qe-21 ~]# hammer  -u admin -p changeme capsule content synchronize --id=2
[........................................................................................................................                     ] [85%]
Error: OST0004: Feed URL not specified
PLP0000: Importer indicated a failed response
[root@cloud-qe-21 ~]# 



on capsule node: I see these errors in /var/log/Jul 19 02:50:03 cloud-qe-4 pulp: pulp_rpm.plugins.importers.yum.parse.treeinfo:ERROR: some distro file downloads failed
Jul 19 02:50:03 cloud-qe-4 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2057-27104)
Jul 19 02:50:03 cloud-qe-4 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2057-27104) Traceback (most recent call last):
Jul 19 02:50:03 cloud-qe-4 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2057-27104)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 228, in run
Jul 19 02:50:03 cloud-qe-4 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2057-27104)     dist_sync.run()
Jul 19 02:50:03 cloud-qe-4 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2057-27104)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/parse/treeinfo.py", line 125, in run
Jul 19 02:50:03 cloud-qe-4 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2057-27104)     self._run(tmp_dir)
Jul 19 02:50:03 cloud-qe-4 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2057-27104)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/parse/treeinfo.py", line 195, in _run
Jul 19 02:50:03 cloud-qe-4 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2057-27104)     for destination, location in downloaded:
Jul 19 02:50:03 cloud-qe-4 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2057-27104)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/parse/treeinfo.py", line 284, in download_files
Jul 19 02:50:03 cloud-qe-4 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2057-27104)     raise DownloadFailed()
Jul 19 02:50:03 cloud-qe-4 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2057-27104) DownloadFailed
Jul 19 02:50:03 cloud-qe-4 pulp: pulp.server.async.tasks:INFO: Task failed : [edd983ce-709c-4d10-bfa6-18dfd1c7e468]
Jul 19 02:50:03 cloud-qe-4 pulp: celery.worker.job:ERROR: (1789-27104) Task pulp.server.managers.repo.sync.sync[edd983ce-709c-4d10-bfa6-18dfd1c7e468] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Jul 19 02:50:03 cloud-qe-4 pulp: celery.worker.job:ERROR: (1789-27104) Traceback (most recent call last):
Jul 19 02:50:03 cloud-qe-4 pulp: celery.worker.job:ERROR: (1789-27104)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Jul 19 02:50:03 cloud-qe-4 pulp: celery.worker.job:ERROR: (1789-27104)     R = retval = fun(*args, **kwargs)
Jul 19 02:50:03 cloud-qe-4 pulp: celery.worker.job:ERROR: (1789-27104)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__
Jul 19 02:50:03 cloud-qe-4 pulp: celery.worker.job:ERROR: (1789-27104)     return super(Task, self).__call__(*args, **kwargs)
Jul 19 02:50:03 cloud-qe-4 pulp: celery.worker.job:ERROR: (1789-27104)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Jul 19 02:50:03 cloud-qe-4 pulp: celery.worker.job:ERROR: (1789-27104)     return super(PulpTask, self).__call__(*args, **kwargs)
Jul 19 02:50:03 cloud-qe-4 pulp: celery.worker.job:ERROR: (1789-27104)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Jul 19 02:50:03 cloud-qe-4 pulp: celery.worker.job:ERROR: (1789-27104)     return self.run(*args, **kwargs)
Jul 19 02:50:03 cloud-qe-4 pulp: celery.worker.job:ERROR: (1789-27104)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 810, in sync
Jul 19 02:50:03 cloud-qe-4 pulp: celery.worker.job:ERROR: (1789-27104)     raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
Jul 19 02:50:03 cloud-qe-4 pulp: celery.worker.job:ERROR: (1789-27104) PulpExecutionException: Importer indicated a failed response
Jul 19 02:50:03 cloud-qe-4 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[c88d6d88-5836-4061-b572-7d81b40accdd] succeeded in 0.00428685999941s: None
messages
Comment 27 Sachin Ghai 2016-07-19 07:25:52 UTC 
On satellite server, under production.log:
==========================================


2016-07-19 02:50:09 [foreman-tasks/action] [E] PLP0000: Importer indicated a failed response (Katello::Errors::PulpError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.64/app/lib/actions/pulp/abstract_async_task.rb:121:in `block in external_task='
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.64/app/lib/actions/pulp/abstract_async_task.rb:119:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.64/app/lib/actions/pulp/abstract_async_task.rb:119:in `external_task='
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/polling.rb:98:in `poll_external_task_with_rescue'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/polling.rb:21:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/cancellable.rb:9:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.64/app/lib/actions/pulp/abstract_async_task.rb:45:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:506:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:30:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
Comment 28 Sachin Ghai 2016-07-19 07:26:25 UTC 
production.log
=================

2016-07-19 03:24:02 [foreman-tasks/action] [E] OST0004: Feed URL not specified (Katello::Errors::PulpError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.64/app/lib/actions/pulp/abstract_async_task.rb:121:in `block in external_task='
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.64/app/lib/actions/pulp/abstract_async_task.rb:119:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.64/app/lib/actions/pulp/abstract_async_task.rb:119:in `external_task='
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/polling.rb:98:in `poll_external_task_with_rescue'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/polling.rb:21:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/cancellable.rb:9:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.64/app/lib/actions/pulp/abstract_async_task.rb:45:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:506:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:30:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.64/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.64/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.64/app/models/katello/concerns/user_extensions.rb:20:in `cp_config'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.64/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.64/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.64/app/lib/actions/middleware/remote_action.rb:16:in `run'
 | /opt/the
Comment 29 Sachin Ghai 2016-07-19 07:27:35 UTC 
on capsule node: /var/log/messages:

Jul 19 03:24:02 cloud-qe-4 pulp: celery.worker.strategy:INFO: Received task: pulp.server.tasks.repository.distributor_update[a333d256-bb8b-44d8-b25b-67c3c38489cd]
Jul 19 03:24:02 cloud-qe-4 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[30784aaf-b9e2-47d3-aff9-aa42a4129445]
Jul 19 03:24:02 cloud-qe-4 pulp: pulp.server.async.tasks:INFO: Task failed : [a333d256-bb8b-44d8-b25b-67c3c38489cd]
Jul 19 03:24:02 cloud-qe-4 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[a99267de-d070-4621-b6e5-c66df96bf632] succeeded in 0.0381358060113s: None
Jul 19 03:24:02 cloud-qe-4 pulp: celery.worker.job:ERROR: (1777-40448) Task pulp.server.tasks.repository.distributor_update[a333d256-bb8b-44d8-b25b-67c3c38489cd] raised unexpected: PulpDataException('Relative path [Default_Organization/Dev/cv_atomic/content/dist/rhel/atomic/7/7Server/x86_64/ostree/repo] for repository [Default_Organization-Dev-cv_atomic-Red_Hat_Enterprise_Linux_Atomic_Host-Red_Hat_Enterprise_Linux_Atomic_Host_Trees] conflicts with existing relative path [Default_Organization/Dev/cv_atomic/content/dist/rhel/atomic/7/7Server/x86_64/ostree/repo] for repository [Default_Organization-Dev-cv_atomic-Red_Hat_Enterprise_Linux_Atomic_Host-Red_Hat_Enterprise_Linux_Atomic_Host_Trees]',)
Jul 19 03:24:02 cloud-qe-4 pulp: celery.worker.job:ERROR: (1777-40448) Traceback (most recent call last):
Jul 19 03:24:02 cloud-qe-4 pulp: celery.worker.job:ERROR: (1777-40448)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Jul 19 03:24:02 cloud-qe-4 pulp: celery.worker.job:ERROR: (1777-40448)     R = retval = fun(*args, **kwargs)
Jul 19 03:24:02 cloud-qe-4 pulp: celery.worker.job:ERROR: (1777-40448)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__
Jul 19 03:24:02 cloud-qe-4 pulp: celery.worker.job:ERROR: (1777-40448)     return super(Task, self).__call__(*args, **kwargs)
Jul 19 03:24:02 cloud-qe-4 pulp: celery.worker.job:ERROR: (1777-40448)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Jul 19 03:24:02 cloud-qe-4 pulp: celery.worker.job:ERROR: (1777-40448)     return super(PulpTask, self).__call__(*args, **kwargs)
Jul 19 03:24:02 cloud-qe-4 pulp: celery.worker.job:ERROR: (1777-40448)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Jul 19 03:24:02 cloud-qe-4 pulp: celery.worker.job:ERROR: (1777-40448)     return self.run(*args, **kwargs)
Jul 19 03:24:02 cloud-qe-4 pulp: celery.worker.job:ERROR: (1777-40448)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/distributor.py", line 239, in update
Jul 19 03:24:02 cloud-qe-4 pulp: celery.worker.job:ERROR: (1777-40448)     raise exceptions.PulpDataException(message)
Jul 19 03:24:02 cloud-qe-4 pulp: celery.worker.job:ERROR: (1777-40448) PulpDataException: Relative path [Default_Organization/Dev/cv_atomic/content/dist/rhel/atomic/7/7Server/x86_64/ostree/repo] for repository [Default_Organization-Dev-cv_atomic-Red_Hat_Enterprise_Linux_Atomic_Host-Red_Hat_Enterprise_Linux_Atomic_Host_Trees] conflicts with existing relative path [Default_Organization/Dev/cv_atomic/content/dist/rhel/atomic/7/7Server/x86_64/ostree/repo] for repository [Default_Organization-Dev-cv_atomic-Red_Hat_Enterprise_Linux_Atomic_Host-Red_Hat_Enterprise_Linux_Atomic_Host_Trees]
Jul 19 03:24:02 cloud-qe-4 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[fb77e299-d8cd-4e4c-ad74-0e38e245a45d]
Comment 30 Sachin Ghai 2016-07-19 14:20:24 UTC 
On capsule installed ostree version and libtasn1

[root@cloud-qe-4 ~]# rpm -qa | grep ostree
ostree-2016.5-3.atomic.el7.x86_64
python-pulp-ostree-common-1.1.1-1.el7sat.noarch
pulp-ostree-plugins-1.1.1-1.el7sat.noarch
[root@cloud-qe-4 ~]# rpm -qa | grep libtasn
libtasn1-3.8-3.el7.x86_64
Comment 31 John Mitsch 2016-07-19 17:49:19 UTC 
Created redmine issue http://projects.theforeman.org/issues/15745 from this bug
Comment 32 Sachin Ghai 2016-07-21 07:56:49 UTC 
Verified sat6.2 snap21 and this time I got slightly different error:

[root@cloud-qe-07 ~]# hammer -u admin -p changeme capsule content synchronize --id=2
[............................................................................................................................................] [100%]
Error: PLP0000: Importer indicated a failed response
OST0005: Fetch summary failed. Reason: GLib.Error('Server required TLS certificate', 'g-tls-error-quark', 5)


==> /var/log/foreman/production.log <==
2016-07-21 03:44:46 [foreman-tasks/action] [E] OST0005: Fetch summary failed. Reason: GLib.Error('Server required TLS certificate', 'g-tls-error-quark', 5) (Katello::Errors::PulpError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.66/app/lib/actions/pulp/abstract_async_task.rb:121:in `block in external_task='
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.66/app/lib/actions/pulp/abstract_async_task.rb:119:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.66/app/lib/actions/pulp/abstract_async_task.rb:119:in `external_task='
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/polling.rb:98:in `poll_external_task_with_rescue'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/polling.rb:21:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/cancellable.rb:9:in `run'
Comment 33 Sachin Ghai 2016-07-21 07:59:27 UTC 
I checked installed packages:

pulp-ostree-plugins-1.1.1-1.el7sat.noarch
python-pulp-ostree-common-1.1.1-1.el7sat.noarch
ostree-2016.5-3.atomic.el7.x86_64
tfm-rubygem-katello_ostree-3.0.0.66-1.el7sat.noarch

[root@cloud-qe-07 ~]# rpm -qa | grep  libtasn1
libtasn1-3.8-2.el7.x86_64

~]# rpm -qa | grep rubygem-katello-3.0.0.66-1
tfm-rubygem-katello-3.0.0.66-1.el7sat.noarch


looks like somehow libtasn1-3.8-2.el7.x86_64 is getting installed instead of libtasn1-3.8-3.

Will upgrade libtasn1-3.8-2 to 3.8-3 and update here.
Comment 34 Sachin Ghai 2016-07-21 08:09:39 UTC 
I Updated following package and restarted the katello-service and still getting following issue

Installed: libtasn1.x86_640:3.8-3.el7                                                                                                                         

[root ~]# hammer -u admin -p changeme capsule content synchronize --id=2
[............................................................................................................................................] [100%]
Error: PLP0000: Importer indicated a failed response
OST0005: Fetch summary failed. Reason: GLib.Error('Server required TLS certificate', 'g-tls-error-quark', 5)

two queries:
1) any idea why ibtasn1-3.8-2 is getting installed ? I see this package is in base rhel7 repo too. But in any case, on fresh setup version 3.8-3 should be installed, if that contain the fix

--
libtasn1-3.8-2.el7.x86_64 : The ASN.1 library used in GNUTLS
Repo        : rhel-7-server-rpms

libtasn1-3.8-3.el7.x86_64 : The ASN.1 library used in GNUTLS
Repo        : sat6
--


2) any other step needs to be performed after upgrading the libtasn1 ?
Comment 35 Sachin Ghai 2016-07-21 08:20:42 UTC 
I re-ran the installer too after updating the libtasn1 package.. but still getting:

OST0005: Fetch summary failed. Reason: GLib.Error('Server required TLS certificate', 'g-tls-error-quark', 5)
Comment 36 John Mitsch 2016-07-21 13:14:43 UTC 
Sachin,

Investigating your setup, I see libtasn1 is not upgraded to libtasn1-3.8-3 on the capsule. Upgrading this from 3.8.2 -> 3.8.3 on the capsule should fix the capsule syncs.

As far as why these updated libtasn1 packages are not included in Satellite, we should track that here https://bugzilla.redhat.com/show_bug.cgi?id=1354608
Comment 37 Sachin Ghai 2016-07-21 14:02:37 UTC 
hey John,

I installed 3.8.3 on the capsule and re-synced contents on capsule but I see this error:

 ~]# hammer -u admin -p changeme capsule content synchronize --id=2
[............................................................................................................................................] [100%]
Error: PLP0000: Importer indicated a failed response


I see capsule contents on capsule too.
[root@cloud-qe-06 content]# ll
total 20
-rw-r--r--.   1 apache apache   38 Jul 21 09:53 config
drwxr-xr-x. 258 apache apache 8192 Jul 21 09:33 objects
drwxr-xr-x.   4 apache apache   32 Jul 21 03:44 refs
drwxr-xr-x.   2 apache apache    6 Jul 21 09:33 state
-rw-r--r--.   1 apache apache  121 Jul 21 09:53 summary
drwxr-xr-x.   4 apache apache   65 Jul 21 09:53 tmp
drwxr-xr-x.   2 apache apache    6 Jul 21 03:44 uncompressed-objects-cache


will re-try on fresh capsule.
Comment 38 Mike McCune 2016-07-22 00:27:36 UTC 
Sachin,

I tested this on a fresh Sat + Capsule installation and ensured that I was fully yum-updated *after* installing the Satellite and Capsule RPMs.

The sync between sat>capsule worked fine, without error.
Comment 39 Mike McCune 2016-07-22 06:00:13 UTC 
Dev Verified on SNAP 21.1
Comment 40 Sachin Ghai 2016-07-22 09:23:14 UTC 
Thanks Mike.

I just installed fresh setup with snap21.1 on rhel7 and can see libtasn1 package is getting updated with installer.

on satellite:
=============

[root@cloud-qe ~]# rpm -qa | grep libtasn
libtasn1-3.8-2.el7.x86_64
[root@cloud-qe-06 ~]# satellite-installer --scenario satellite --foreman-admin-password changeme --katello-enable-ostree=true
Installing             Done                                               [100%] [..................................................................]
  Success!
 
      capsule-certs-generate --capsule-fqdn "$CAPSULE" --certs-tar "~/$CAPSULE-certs.tar"


[root@cloud-qe ~]# rpm -qa | grep libtasn
libtasn1-3.8-3.el7.x86_64


on capsule too. Its getting updated by installer:

[root@cloud-qe ~]# rpm -qa | grep libtasn1
libtasn1-3.8-2.el7.x86_64


[root@cloud-qe- ~]# satellite-installer --scenario capsule --capsule-enable-ostree true
Installing             Done                                               [100%] [..................................................................]
  Success!
  The full log is at /var/log/foreman-installer/capsule.log
[root@cloud-qe ~]# rpm -qa | grep libtasn1
libtasn1-3.8-3.el7.x86_64
Comment 41 Sachin Ghai 2016-07-22 10:07:58 UTC 
Ostree contents successfully synced on capsule

[root@cloud-qe ~]# hammer -u admin -p changeme capsule content synchronize --id=2
[............................................................................................................................................] [100%]
[r

contents are visible on capsule:

[root@cloud-qe ~]# cd /var/lib/pulp/content/shared/ostree/afb9cde17159b594acd55d3e31ae1f3d700f485f880fae77b325d6d47404f990/content/
[root@cloud-qe-21 content]# ll
total 20
-rw-r--r--.   1 apache apache   38 Jul 22 06:05 config
drwxr-xr-x. 258 apache apache 8192 Jul 22 06:05 objects
drwxr-xr-x.   4 apache apache   32 Jul 22 06:04 refs
drwxr-xr-x.   2 apache apache    6 Jul 22 06:05 state
-rw-r--r--.   1 apache apache  121 Jul 22 06:05 summary
drwxr-xr-x.   4 apache apache   65 Jul 22 06:05 tmp
drwxr-xr-x.   2 apache apache    6 Jul 22 06:04 uncompressed-objects-cache
[root@cloud-qe content]# pwd
/var/lib/pulp/content/shared/ostree/afb9cde17159b594acd55d3e31ae1f3d700f485f880fae77b325d6d47404f990/content
Comment 42 Bryan Kearney 2016-07-27 11:13:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501