Bug 1312640

Summary: hammer repo export throws error: unknown file type:
Product: Red Hat Satellite 6 Reporter: Mike McCune <mmccune>
Component: Content ManagementAssignee: Chris Duryee <cduryee>
Status: CLOSED ERRATA QA Contact: Jitendra Yejare <jyejare>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2.0CC: bbuckingham, bkearney, cwelton, ehelms, jyejare, lpramuk, omaciel, sthirugn, swadeley
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
URL: http://projects.theforeman.org/issues/13781
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-27 09:24:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
Settings for exporting
none
Stacktrace generated when exporting a repository via hammer none

Description Mike McCune 2016-02-28 12:03:37 UTC
1) Create and Sync (immediate) a repo
2) Export:

# hammer repository export --id 32
[...........................................] [100%]
unknown file type: /var/lib/pulp/published/yum/master/group_export_distributor/Default_Organization-Test_Product-synced-repo

Comment 2 Chris Duryee 2016-02-29 14:19:38 UTC
This appears to be http://projects.theforeman.org/issues/13781 which is fixed in upstream.


# hammer repository export --id 32
Ignoring ruby-libvirt-0.5.2 because its extensions are not built.  Try: gem pristine ruby-libvirt --version 0.5.2
[.................................................................................................................................] [100%]
unknown file type: /var/lib/pulp/published/yum/master/group_export_distributor/Default_Organization-Test_Product-synced-repo

[root@sat-r220-06 ~]# setenforce 0

[root@sat-r220-06 ~]# hammer repository export --id 32
Ignoring ruby-libvirt-0.5.2 because its extensions are not built.  Try: gem pristine ruby-libvirt --version 0.5.2
[.................................................................................................................................] [100%]


*************************

# audit2allow -a


#============= passenger_t ==============
allow passenger_t httpd_sys_rw_content_t:dir { read search open getattr };
allow passenger_t httpd_sys_rw_content_t:file { read getattr open ioctl };

#============= streamer_t ==============
allow streamer_t tmp_t:dir write;

************************

NOTE: the streamer_t denial is not related to export. I believe it's an unrelated issue.

Comment 3 Bryan Kearney 2016-02-29 15:09:25 UTC
Upstream bug component is Content Management

Comment 4 Bryan Kearney 2016-02-29 15:09:27 UTC
Moving to POST since upstream bug http://projects.theforeman.org/issues/13781 has been closed

Comment 6 Og Maciel 2016-03-18 20:14:23 UTC
[root@ibm-x3250m4-01 ~]# LANG=en_US.UTF-8 hammer -v -u admin -p changeme  repository export --id 1
[ERROR 2016-03-18 16:13:51 Exception] ERF42-3196 [Foreman::Exception]: Unable to export, 'pulp_export_destination' setting is not set to a valid directory.
Could not export the repository:
  ERF42-3196 [Foreman::Exception]: Unable to export, 'pulp_export_destination' setting is not set to a valid directory.
[ERROR 2016-03-18 16:13:51 Exception]

RestClient::InternalServerError (500 Internal Server Error):
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:230:in `process_result'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:178:in `block in transmit'
    /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:853:in `start'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:172:in `transmit'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:67:in `post'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.0.14/lib/apipie_bindings/api.rb:286:in `call_client'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.0.14/lib/apipie_bindings/api.rb:217:in `http_call'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.0.14/lib/apipie_bindings/api.rb:162:in `call'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.0.14/lib/apipie_bindings/resource.rb:14:in `call'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.5.1.3/lib/hammer_cli/apipie/command.rb:43:in `send_request'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_foreman-0.5.1.2/lib/hammer_cli_foreman/commands.rb:189:in `send_request'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_foreman_tasks-0.0.10/lib/hammer_cli_foreman_tasks/async_command.rb:14:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/command.rb:68:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.5.1.3/lib/hammer_cli/abstract.rb:22:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/subcommand/execution.rb:11:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/command.rb:68:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.5.1.3/lib/hammer_cli/abstract.rb:22:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/subcommand/execution.rb:11:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/command.rb:68:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.5.1.3/lib/hammer_cli/abstract.rb:22:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/command.rb:133:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.5.1.3/bin/hammer:125:in `<top (required)>'
    /usr/bin/hammer:23:in `load'
    /usr/bin/hammer:23:in `<main>'

Comment 7 Og Maciel 2016-03-18 20:30:08 UTC
Created attachment 1137866 [details]
Settings for exporting

It is necessary to first define a path (writable by the apache user) for the 'pulp_export_destination' setting, as shown here.

Comment 8 Og Maciel 2016-03-18 20:48:28 UTC
To verify this issue:

* Update your 'pulp_export_destination' setting and set it to "/var/www/html/pub" (web UI, Settings menu, search for pulp_export_destination)
* Create a repository (I used an existing, synchronized RHEL repo) and sync it
* Use the hammer repository export command as per first comment here

Comment 9 Og Maciel 2016-03-18 20:49:19 UTC
Created attachment 1137881 [details]
Stacktrace generated when exporting a repository via hammer

Comment 10 Og Maciel 2016-03-18 20:50:15 UTC
Mike McCune also could not test this feature, so I am failing it.

Tested against Satellite 6.2.0 SNAP 4.0 build

Comment 11 Chris Duryee 2016-03-21 18:14:28 UTC
There are a few issues found:

1. there was a missing cherry-pick for katello-selinux which will be in the next snap. I ran "audit2allow -a" and saw "allow passenger_t httpd_sys_rw_content_t:dir search;". After updating the selinux policy, you should see a message like "#!!!! This avc is allowed in the current policy" which indicates that selinux is copacetic. Note that fresh installs that didnt have a prior denial here will not have any message, it will just work:)

2. the directory needs to be owned by foreman user and group, not apache. This is documented in the upstream docs at http://www.katello.org/docs//user_guide/disconnected/, downstream docs are being updated for this (https://bugzilla.redhat.com/show_bug.cgi?id=1285244#c50 and https://bugzilla.redhat.com/show_bug.cgi?id=1285244#c55) I think it was apache.apache in the past, probably on one of the etherpads at one point.

3. default selinux policy does not allow passenger to write to /var/www/html/*, even if the file-level permissions are correct.

Typically I export repos to /mnt/exports, so I can pretend that I'm exporting to a mount that can be shared. I don't know how common it will be to export to /var/www/html/pub for the disconnected use case, since it short-circuits the "export, walk it over, then import" flow.

Having said that, we can add an additional selinux rule to allow issue 3 if you think it would be helpful. It would be impossible to guess all the places someone would export and create rules in advance, so maybe a kbase article would be more helpful that shows how to do it generically.

Marking bz as NEEDSINFO on omaciel for to get feedback on third issue.

Comment 12 Chris Duryee 2016-03-21 18:25:26 UTC
Something I should have mentioned, for issue 3 in comment #11 the selinux denial only affects certain directories like /var/www/html/. Dirs like /mnt/export should be unaffected and do not require additional rules.

Comment 13 Og Maciel 2016-03-21 18:36:16 UTC
Hi Chris,

[root@ibm-x3250m4-01 pub]# mkdir /mnt/export
[root@ibm-x3250m4-01 pub]# chown foreman.foreman /mnt/export
[root@ibm-x3250m4-01 pub]# ls -l /mnt/export/
total 0
[root@ibm-x3250m4-01 pub]# ls -ld /mnt/export/
drwxr-xr-x. 2 foreman foreman 6 Mar 21 14:34 /mnt/export/
[root@ibm-x3250m4-01 pub]# LANG=en_US.UTF-8 hammer -v -u admin -p changeme  repository export --id 1
[ERROR 2016-03-21 14:35:06 Exception] ERF42-6337 [Foreman::Exception]: Unable to export, 'pulp_export_destination' setting is not a writable directory.
Could not export the repository:
  ERF42-6337 [Foreman::Exception]: Unable to export, 'pulp_export_destination' setting is not a writable directory.
[ERROR 2016-03-21 14:35:06 Exception]

RestClient::InternalServerError (500 Internal Server Error):
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:230:in `process_result'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:178:in `block in transmit'
    /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:853:in `start'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:172:in `transmit'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:67:in `post'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.0.14/lib/apipie_bindings/api.rb:286:in `call_client'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.0.14/lib/apipie_bindings/api.rb:217:in `http_call'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.0.14/lib/apipie_bindings/api.rb:162:in `call'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.0.14/lib/apipie_bindings/resource.rb:14:in `call'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.5.1.3/lib/hammer_cli/apipie/command.rb:43:in `send_request'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_foreman-0.5.1.2/lib/hammer_cli_foreman/commands.rb:189:in `send_request'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_foreman_tasks-0.0.10/lib/hammer_cli_foreman_tasks/async_command.rb:14:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/command.rb:68:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.5.1.3/lib/hammer_cli/abstract.rb:22:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/subcommand/execution.rb:11:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/command.rb:68:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.5.1.3/lib/hammer_cli/abstract.rb:22:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/subcommand/execution.rb:11:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/command.rb:68:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.5.1.3/lib/hammer_cli/abstract.rb:22:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/command.rb:133:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.5.1.3/bin/hammer:125:in `<top (required)>'
    /usr/bin/hammer:23:in `load'
    /usr/bin/hammer:23:in `<main>'

By the way, I think that a KB article would be great for those who like myself are struggling a bit :)

Comment 16 Chris Duryee 2016-03-28 14:24:17 UTC
Og,

I created https://bugzilla.redhat.com/show_bug.cgi?id=1321589 so you don't have to do the steps noted in issue 3. Jitendra hit it as well, so IMO it is a common enough use case to warrant adding a rule.

Comment 18 Og Maciel 2016-04-04 15:37:55 UTC
QE: I have been able to test this issue using a small YUM repo but had issues when exporting a large-ish Red Hat repository. To properly verify this issue I think we need to:

* Test exporting Background download policy repos
* Test exporting OnDemand download policy repos
* Verify https://bugzilla.redhat.com/show_bug.cgi?id=1323730

Comment 24 Lukas Pramuk 2016-04-25 13:47:19 UTC
New BZ to raise a correct error message when exporting individual non-yum repos
 BZ #1330166

Comment 26 errata-xmlrpc 2016-07-27 09:24:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501