Bug 1009114

Summary: Puppet Install Distributor should create target directory if possible
Product: [Retired] Pulp Reporter: Mike McCune <mmccune>
Component: puppet-supportAssignee: Jeff Ortel <jortel>
Status: CLOSED CURRENTRELEASE QA Contact: Preethi Thomas <pthomas>
Severity: medium Docs Contact:
Priority: high    
Version: MasterCC: jortel, mhrivnak, pthomas
Target Milestone: ---Keywords: Triaged
Target Release: 2.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-12-09 14:30:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Mike McCune 2013-09-17 17:42:19 UTC 
We are using the Puppet Install Distributor with Katello to publish and manage the puppet module repositories and make them available to the Puppet Master.

We are hitting the issue that we can't publish any of our repositories without first creating the directory path that Pulp expects to publish to.

The issue is that the katello user on the system doesn't have the ability to create and set access permission on the large numbers of directories we are going to need to maintain for all the repos managed by katello, eg:


/etc/puppet/ACME_Corporation-puppet-test-puppet1
/etc/puppet/ACME_Corporation-puppet-test-puppet2
/etc/puppet/ACME_Corporation-puppet-test-puppet3
/etc/puppet/ACME_Corporation-Library-pview1-puppet-test-puppet6
/etc/puppet/ACME_Corporation-Dev-pview1-puppet-test-puppet6
/etc/puppet/ACME_Corporation-Test-pview1-puppet-test-puppet6
/etc/puppet/ACME_Corporation-Production-pview2-puppet-test-puppet6

since these directories are calculated and generated based on the repoid this isn't something we can pre-create during install time.

We can possibly publish to a different directory outside of /etc/puppet but to do this we would want to pick a location that apache had write access to.  Likewise since Pulp is running in apache it would be highly advantageous for Pulp to actually create the directories during the publish routine.

So this is a request to see if we can get Pulp to create these directories during publish time and if so, we can adapt Katello to specify the correct location to do so.
Comment 1 Mike McCune 2013-09-17 18:14:08 UTC 
NOTE: I do realize the docs do state that it is the callers responsibility to create and ensure access:

http://pulp-puppet-dev-guide.readthedocs.org/en/latest/plugin_conf.html#install-distributor

"install_path
    Full path to the directory where modules should be installed. It is the user’s responsibility to ensure that Pulp can write to this directory. "

but after working with the distributor a bit we are still requesting that Pulp at least attempt to create the directory in the path specified.  Katello/Foreman can ensure that the parent directory exists and that apache has write access, eg:

-> /etc/puppet : can be written to by apache
-> /etc/puppet/ACME_Corporation-puppet-test-puppet1 : created by Pulp
Comment 2 Jeff Ortel 2013-09-17 21:21:55 UTC 
https://github.com/pulp/pulp_puppet/pull/74
Comment 3 Jeff Ortel 2013-09-18 23:58:44 UTC 
build: 2.3.0-0.14.alpha
Comment 4 Preethi Thomas 2013-09-24 16:00:52 UTC 
Looks like this is fails-qa
  u'summary': u"failed to create destination directory: [Errno 13] Permission denied: '/etc/puppet/puppet-copy/'",


   u'traceback': None}])
root@pulp-v2-server ~]# ipython
Python 2.6.6 (r266:84292, Oct 12 2012, 14:23:48) 
Type "copyright", "credits" or "license" for more information.

IPython 0.10 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]: from pulp.common import pic

In [2]: pic.connect()

In [3]: pic.POST('/v2/repositories/puppet-copy/distributors/', {'distributor_type_id':'puppet_install_distributor', 'distributor_id':'puppet_install_distributor', 'auto_publish': False, 'distributor_config': {'install_path':'/etc/puppet/puppet-copy/'}})
Request Body
{
  "distributor_id": "puppet_install_distributor", 
  "distributor_type_id": "puppet_install_distributor", 
  "distributor_config": {
    "install_path": "/etc/puppet/puppet-copy/"
  }, 
  "auto_publish": false
}
Response Body
{
  "repo_id": "puppet-copy", 
  "_ns": "repo_distributors", 
  "last_publish": null, 
  "auto_publish": false, 
  "scheduled_publishes": [], 
  "distributor_type_id": "puppet_install_distributor", 
  "scratchpad": null, 
  "_id": {
    "$oid": "5241c7ffd669f040ff0003c6"
  }, 
  "config": {
    "install_path": "/etc/puppet/puppet-copy/"
  }, 
  "id": "puppet_install_distributor"
}
Out[3]: 
(201,
 {u'_id': {u'$oid': u'5241c7ffd669f040ff0003c6'},
  u'_ns': u'repo_distributors',
  u'auto_publish': False,
  u'config': {u'install_path': u'/etc/puppet/puppet-copy/'},
  u'distributor_type_id': u'puppet_install_distributor',
  u'id': u'puppet_install_distributor',
  u'last_publish': None,
  u'repo_id': u'puppet-copy',
  u'scheduled_publishes': [],
  u'scratchpad': None})

In [4]: pic.POST('/v2/repositories/puppet-copy/actions/publish/', {'id':'puppet_install_distributor'})
Request Body
{
  "id": "puppet_install_distributor"
}
Response Body
{
  "task_group_id": null, 
  "call_request_id": "6ebe2022-591a-4eb5-aee4-e673f06206c6", 
  "exception": null, 
  "_href": "/pulp/api/v2/tasks/6ebe2022-591a-4eb5-aee4-e673f06206c6/", 
  "task_id": "6ebe2022-591a-4eb5-aee4-e673f06206c6", 
  "call_request_tags": [
    "pulp:repository:puppet-copy", 
    "pulp:action:publish"
  ], 
  "reasons": [], 
  "start_time": null, 
  "traceback": null, 
  "schedule_id": null, 
  "finish_time": null, 
  "state": "waiting", 
  "result": null, 
  "dependency_failures": {}, 
  "call_request_group_id": null, 
  "progress": {}, 
  "principal_login": "admin", 
  "response": "accepted", 
  "tags": [
    "pulp:repository:puppet-copy", 
    "pulp:action:publish"
  ]
}
Out[4]: 
(202,
 {u'_href': u'/pulp/api/v2/tasks/6ebe2022-591a-4eb5-aee4-e673f06206c6/',
  u'call_request_group_id': None,
  u'call_request_id': u'6ebe2022-591a-4eb5-aee4-e673f06206c6',
  u'call_request_tags': [u'pulp:repository:puppet-copy',
                         u'pulp:action:publish'],
  u'dependency_failures': {},
  u'exception': None,
  u'finish_time': None,
  u'principal_login': u'admin',
  u'progress': {},
  u'reasons': [],
  u'response': u'accepted',
  u'result': None,
  u'schedule_id': None,
  u'start_time': None,
  u'state': u'waiting',
  u'tags': [u'pulp:repository:puppet-copy', u'pulp:action:publish'],
  u'task_group_id': None,
  u'task_id': u'6ebe2022-591a-4eb5-aee4-e673f06206c6',
  u'traceback': None})

In [5]: pic.GET('/v2/repositories/puppet-copy/history/publish/puppet_install_distributor/')
Response Body
[
  {
    "result": "failed", 
    "exception": null, 
    "repo_id": "puppet-copy", 
    "started": "2013-09-24T13:12:59-04:00", 
    "_ns": "repo_publish_results", 
    "completed": "2013-09-24T13:12:59-04:00", 
    "traceback": null, 
    "distributor_type_id": "puppet_install_distributor", 
    "_id": {
      "$oid": "5241c81bd669f040ff0003d0"
    }, 
    "summary": "failed to create destination directory: [Errno 13] Permission denied: '/etc/puppet/puppet-copy/'", 
    "error_message": null, 
    "distributor_id": "puppet_install_distributor", 
    "id": "5241c81bd669f040ff0003d0", 
    "details": {
      "errors": [], 
      "success_unit_keys": []
    }
  }
]
Out[5]: 
(200,
 [{u'_id': {u'$oid': u'5241c81bd669f040ff0003d0'},
   u'_ns': u'repo_publish_results',
   u'completed': u'2013-09-24T13:12:59-04:00',
   u'details': {u'errors': [], u'success_unit_keys': []},
   u'distributor_id': u'puppet_install_distributor',
   u'distributor_type_id': u'puppet_install_distributor',
   u'error_message': None,
   u'exception': None,
   u'id': u'5241c81bd669f040ff0003d0',
   u'repo_id': u'puppet-copy',
   u'result': u'failed',
   u'started': u'2013-09-24T13:12:59-04:00',
   u'summary': u"failed to create destination directory: [Errno 13] Permission denied: '/etc/puppet/puppet-copy/'",
   u'traceback': None}])

In [6]:
Comment 5 Michael Hrivnak 2013-09-26 20:06:45 UTC 
This looks like normal behavior to me, unless you have reason to believe that apache does in fact have permission to create that directory. By default, apache cannot write in /etc/puppet/. The user is responsible for making sure that apache has permission to write in the destination directory, and that's stated in the documentation for the distributor. See the very last sentence here:

http://pulp-puppet-dev-guide.readthedocs.org/en/latest/plugin_conf.html#install-distributor
Comment 6 Jeff Ortel 2013-09-27 14:24:26 UTC 
Agreed.  Moving back to ON_QA for re-test.
Comment 7 Preethi Thomas 2013-10-04 14:57:53 UTC 
verified
[root@pulp-v2-server ~]# rpm -qa pulp-server
pulp-server-2.3.0-0.17.beta.el6.noarch
[root@pulp-v2-server ~]# 


In [1]: from pulp.common import pic

In [2]: pic.connect()

In [3]: pic.POST('/v2/repositories/puppet-copy/distributors/', {'distributor_type_id':'puppet_install_distributor', 'distributor_id':'puppet_install_distributor', 'auto_publish': False, 'distributor_config': {'install_path':'/etc/puppet/puppet-copy/'}})
Request Body
{
  "distributor_id": "puppet_install_distributor", 
  "distributor_type_id": "puppet_install_distributor", 
  "distributor_config": {
    "install_path": "/etc/puppet/puppet-copy/"
  }, 
  "auto_publish": false
}
Response Body
{
  "repo_id": "puppet-copy", 
  "_ns": "repo_distributors", 
  "last_publish": null, 
  "auto_publish": false, 
  "scheduled_publishes": [], 
  "distributor_type_id": "puppet_install_distributor", 
  "scratchpad": null, 
  "_id": {
    "$oid": "524ed515d669f0087a001df1"
  }, 
  "config": {
    "install_path": "/etc/puppet/puppet-copy/"
  }, 
  "id": "puppet_install_distributor"
}
Out[3]: 
(201,
 {u'_id': {u'$oid': u'524ed515d669f0087a001df1'},
  u'_ns': u'repo_distributors',
  u'auto_publish': False,
  u'config': {u'install_path': u'/etc/puppet/puppet-copy/'},
  u'distributor_type_id': u'puppet_install_distributor',
  u'id': u'puppet_install_distributor',
  u'last_publish': None,
  u'repo_id': u'puppet-copy',
  u'scheduled_publishes': [],
  u'scratchpad': None})

In [4]: pic.POST('/v2/repositories/puppet-copy/actions/publish/', {'id':'puppet_install_distributor'})
Request Body
{
  "id": "puppet_install_distributor"
}
Response Body
{
  "task_group_id": null, 
  "call_request_id": "5cc6288e-153f-4053-b9ae-f1e259ae141a", 
  "exception": null, 
  "_href": "/pulp/api/v2/tasks/5cc6288e-153f-4053-b9ae-f1e259ae141a/", 
  "task_id": "5cc6288e-153f-4053-b9ae-f1e259ae141a", 
  "call_request_tags": [
    "pulp:repository:puppet-copy", 
    "pulp:action:publish"
  ], 
  "reasons": [], 
  "start_time": null, 
  "traceback": null, 
  "schedule_id": null, 
  "finish_time": null, 
  "state": "waiting", 
  "result": null, 
  "dependency_failures": {}, 
  "call_request_group_id": null, 
  "progress": {}, 
  "principal_login": "admin", 
  "response": "accepted", 
  "tags": [
    "pulp:repository:puppet-copy", 
    "pulp:action:publish"
  ]
}
Out[4]: 
(202,
 {u'_href': u'/pulp/api/v2/tasks/5cc6288e-153f-4053-b9ae-f1e259ae141a/',
  u'call_request_group_id': None,
  u'call_request_id': u'5cc6288e-153f-4053-b9ae-f1e259ae141a',
  u'call_request_tags': [u'pulp:repository:puppet-copy',
                         u'pulp:action:publish'],
  u'dependency_failures': {},
  u'exception': None,
  u'finish_time': None,
  u'principal_login': u'admin',
  u'progress': {},
  u'reasons': [],
  u'response': u'accepted',
  u'result': None,
  u'schedule_id': None,
  u'start_time': None,
  u'state': u'waiting',
  u'tags': [u'pulp:repository:puppet-copy', u'pulp:action:publish'],
  u'task_group_id': None,
  u'task_id': u'5cc6288e-153f-4053-b9ae-f1e259ae141a',
  u'traceback': None})

In [5]: pic.GET('/v2/repositories/puppet-copy/history/publish/puppet_install_distributor/')
Response Body
[
  {
    "result": "failed", 
    "exception": null, 
    "repo_id": "puppet-copy", 
    "started": "2013-10-04T10:48:19-04:00", 
    "_ns": "repo_publish_results", 
    "completed": "2013-10-04T10:48:20-04:00", 
    "traceback": null, 
    "distributor_type_id": "puppet_install_distributor", 
    "_id": {
      "$oid": "524ed534d669f0087a001dfb"
    }, 
    "summary": "failed to create destination directory: [Errno 13] Permission denied: '/etc/puppet/puppet-copy/'", 
    "error_message": null, 
    "distributor_id": "puppet_install_distributor", 
    "id": "524ed534d669f0087a001dfb", 
    "details": {
      "errors": [], 
      "success_unit_keys": []
    }
  }
]
Out[5]: 
(200,
 [{u'_id': {u'$oid': u'524ed534d669f0087a001dfb'},
   u'_ns': u'repo_publish_results',
   u'completed': u'2013-10-04T10:48:20-04:00',
   u'details': {u'errors': [], u'success_unit_keys': []},
   u'distributor_id': u'puppet_install_distributor',
   u'distributor_type_id': u'puppet_install_distributor',
   u'error_message': None,
   u'exception': None,
   u'id': u'524ed534d669f0087a001dfb',
   u'repo_id': u'puppet-copy',
   u'result': u'failed',
   u'started': u'2013-10-04T10:48:19-04:00',
   u'summary': u"failed to create destination directory: [Errno 13] Permission denied: '/etc/puppet/puppet-copy/'",
   u'traceback': None}])

In [6]: 

In [7]: 

In [8]: pic.POST('/v2/repositories/puppet-copy/distributors/', {'distributor_type_id':'puppet_install_distributor', 'distributor_id':'puppet_install_distributor', 'auto_publish': False, 'distributor_config': {'install_path':'/tmp/preethi-puppet'}})
Request Body
{
  "distributor_id": "puppet_install_distributor", 
  "distributor_type_id": "puppet_install_distributor", 
  "distributor_config": {
    "install_path": "/tmp/preethi-puppet"
  }, 
  "auto_publish": false
}
Response Body
{
  "repo_id": "puppet-copy", 
  "_ns": "repo_distributors", 
  "last_publish": null, 
  "auto_publish": false, 
  "scheduled_publishes": [], 
  "distributor_type_id": "puppet_install_distributor", 
  "scratchpad": null, 
  "_id": {
    "$oid": "524ed597d669f0087a001e06"
  }, 
  "config": {
    "install_path": "/tmp/preethi-puppet"
  }, 
  "id": "puppet_install_distributor"
}
Out[8]: 
(201,
 {u'_id': {u'$oid': u'524ed597d669f0087a001e06'},
  u'_ns': u'repo_distributors',
  u'auto_publish': False,
  u'config': {u'install_path': u'/tmp/preethi-puppet'},
  u'distributor_type_id': u'puppet_install_distributor',
  u'id': u'puppet_install_distributor',
  u'last_publish': None,
  u'repo_id': u'puppet-copy',
  u'scheduled_publishes': [],
  u'scratchpad': None})

In [9]: pic.POST('/v2/repositories/puppet-copy/actions/publish/', {'id':'puppet_install_distributor'})
Request Body
{
  "id": "puppet_install_distributor"
}
Response Body
{
  "task_group_id": null, 
  "call_request_id": "1884fdf1-6ce3-4d2b-8c2d-2ab93b924b25", 
  "exception": null, 
  "_href": "/pulp/api/v2/tasks/1884fdf1-6ce3-4d2b-8c2d-2ab93b924b25/", 
  "task_id": "1884fdf1-6ce3-4d2b-8c2d-2ab93b924b25", 
  "call_request_tags": [
    "pulp:repository:puppet-copy", 
    "pulp:action:publish"
  ], 
  "reasons": [], 
  "start_time": null, 
  "traceback": null, 
  "schedule_id": null, 
  "finish_time": null, 
  "state": "waiting", 
  "result": null, 
  "dependency_failures": {}, 
  "call_request_group_id": null, 
  "progress": {}, 
  "principal_login": "admin", 
  "response": "accepted", 
  "tags": [
    "pulp:repository:puppet-copy", 
    "pulp:action:publish"
  ]
}
Out[9]: 
(202,
 {u'_href': u'/pulp/api/v2/tasks/1884fdf1-6ce3-4d2b-8c2d-2ab93b924b25/',
  u'call_request_group_id': None,
  u'call_request_id': u'1884fdf1-6ce3-4d2b-8c2d-2ab93b924b25',
  u'call_request_tags': [u'pulp:repository:puppet-copy',
                         u'pulp:action:publish'],
  u'dependency_failures': {},
  u'exception': None,
  u'finish_time': None,
  u'principal_login': u'admin',
  u'progress': {},
  u'reasons': [],
  u'response': u'accepted',
  u'result': None,
  u'schedule_id': None,
  u'start_time': None,
  u'state': u'waiting',
  u'tags': [u'pulp:repository:puppet-copy', u'pulp:action:publish'],
  u'task_group_id': None,
  u'task_id': u'1884fdf1-6ce3-4d2b-8c2d-2ab93b924b25',
  u'traceback': None})

In [10]: pic.GET('/v2/repositories/puppet-copy/history/publish/puppet_install_distributor/')
Response Body
[
  {
    "result": "success", 
    "exception": null, 
    "repo_id": "puppet-copy", 
    "started": "2013-10-04T10:50:07-04:00", 
    "_ns": "repo_publish_results", 
    "completed": "2013-10-04T10:50:07-04:00", 
    "traceback": null, 
    "distributor_type_id": "puppet_install_distributor", 
    "_id": {
      "$oid": "524ed59fd669f0087a001e10"
    }, 
    "summary": "success", 
    "error_message": null, 
    "distributor_id": "puppet_install_distributor", 
    "id": "524ed59fd669f0087a001e10", 
    "details": {
      "errors": [], 
      "success_unit_keys": [
        {
          "version": "4.1.0", 
          "name": "stdlib", 
          "author": "puppetlabs"
        }
      ]
    }
  }, 
  {
    "result": "failed", 
    "exception": null, 
    "repo_id": "puppet-copy", 
    "started": "2013-10-04T10:48:19-04:00", 
    "_ns": "repo_publish_results", 
    "completed": "2013-10-04T10:48:20-04:00", 
    "traceback": null, 
    "distributor_type_id": "puppet_install_distributor", 
    "_id": {
      "$oid": "524ed534d669f0087a001dfb"
    }, 
    "summary": "failed to create destination directory: [Errno 13] Permission denied: '/etc/puppet/puppet-copy/'", 
    "error_message": null, 
    "distributor_id": "puppet_install_distributor", 
    "id": "524ed534d669f0087a001dfb", 
    "details": {
      "errors": [], 
      "success_unit_keys": []
    }
  }
]
Out[10]: 
(200,
 [{u'_id': {u'$oid': u'524ed59fd669f0087a001e10'},
   u'_ns': u'repo_publish_results',
   u'completed': u'2013-10-04T10:50:07-04:00',
   u'details': {u'errors': [],
                u'success_unit_keys': [{u'author': u'puppetlabs',
                                        u'name': u'stdlib',
                                        u'version': u'4.1.0'}]},
   u'distributor_id': u'puppet_install_distributor',
   u'distributor_type_id': u'puppet_install_distributor',
   u'error_message': None,
   u'exception': None,
   u'id': u'524ed59fd669f0087a001e10',
   u'repo_id': u'puppet-copy',
   u'result': u'success',
   u'started': u'2013-10-04T10:50:07-04:00',
   u'summary': u'success',
   u'traceback': None},
  {u'_id': {u'$oid': u'524ed534d669f0087a001dfb'},
   u'_ns': u'repo_publish_results',
   u'completed': u'2013-10-04T10:48:20-04:00',
   u'details': {u'errors': [], u'success_unit_keys': []},
   u'distributor_id': u'puppet_install_distributor',
   u'distributor_type_id': u'puppet_install_distributor',
   u'error_message': None,
   u'exception': None,
   u'id': u'524ed534d669f0087a001dfb',
   u'repo_id': u'puppet-copy',
   u'result': u'failed',
   u'started': u'2013-10-04T10:48:19-04:00',
   u'summary': u"failed to create destination directory: [Errno 13] Permission denied: '/etc/puppet/puppet-copy/'",
   u'traceback': None}])


[root@pulp-v2-server ~]# ls -l /tmp/preethi-puppet/
total 4
drwxr-xr-x. 6 apache apache 4096 May 13 10:53 stdlib
[root@pulp-v2-server ~]#
Comment 8 Preethi Thomas 2013-12-09 14:30:33 UTC 
Pulp 2.3 released.