Bug 1312640 - hammer repo export throws error: unknown file type:
hammer repo export throws error: unknown file type:
Status: CLOSED ERRATA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Content Management (Show other bugs)
6.2.0
Unspecified Unspecified
unspecified Severity high (vote)
: Beta
: --
Assigned To: Chris Duryee
Jitendra Yejare
http://projects.theforeman.org/issues...
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-28 07:03 EST by Mike McCune
Modified: 2016-07-27 05:24 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-07-27 05:24:05 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Settings for exporting (16.55 KB, image/png)
2016-03-18 16:30 EDT, Og Maciel
no flags Details
Stacktrace generated when exporting a repository via hammer (57.50 KB, text/plain)
2016-03-18 16:49 EDT, Og Maciel
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Foreman Issue Tracker 13781 None None None 2016-04-22 12:28 EDT

  None (edit)
Description Mike McCune 2016-02-28 07:03:37 EST
1) Create and Sync (immediate) a repo
2) Export:

# hammer repository export --id 32
[...........................................] [100%]
unknown file type: /var/lib/pulp/published/yum/master/group_export_distributor/Default_Organization-Test_Product-synced-repo
Comment 2 Chris Duryee 2016-02-29 09:19:38 EST
This appears to be http://projects.theforeman.org/issues/13781 which is fixed in upstream.


# hammer repository export --id 32
Ignoring ruby-libvirt-0.5.2 because its extensions are not built.  Try: gem pristine ruby-libvirt --version 0.5.2
[.................................................................................................................................] [100%]
unknown file type: /var/lib/pulp/published/yum/master/group_export_distributor/Default_Organization-Test_Product-synced-repo

[root@sat-r220-06 ~]# setenforce 0

[root@sat-r220-06 ~]# hammer repository export --id 32
Ignoring ruby-libvirt-0.5.2 because its extensions are not built.  Try: gem pristine ruby-libvirt --version 0.5.2
[.................................................................................................................................] [100%]


*************************

# audit2allow -a


#============= passenger_t ==============
allow passenger_t httpd_sys_rw_content_t:dir { read search open getattr };
allow passenger_t httpd_sys_rw_content_t:file { read getattr open ioctl };

#============= streamer_t ==============
allow streamer_t tmp_t:dir write;

************************

NOTE: the streamer_t denial is not related to export. I believe it's an unrelated issue.
Comment 3 Bryan Kearney 2016-02-29 10:09:25 EST
Upstream bug component is Content Management
Comment 4 Bryan Kearney 2016-02-29 10:09:27 EST
Moving to POST since upstream bug http://projects.theforeman.org/issues/13781 has been closed
Comment 6 Og Maciel 2016-03-18 16:14:23 EDT
[root@ibm-x3250m4-01 ~]# LANG=en_US.UTF-8 hammer -v -u admin -p changeme  repository export --id 1
[ERROR 2016-03-18 16:13:51 Exception] ERF42-3196 [Foreman::Exception]: Unable to export, 'pulp_export_destination' setting is not set to a valid directory.
Could not export the repository:
  ERF42-3196 [Foreman::Exception]: Unable to export, 'pulp_export_destination' setting is not set to a valid directory.
[ERROR 2016-03-18 16:13:51 Exception]

RestClient::InternalServerError (500 Internal Server Error):
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:230:in `process_result'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:178:in `block in transmit'
    /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:853:in `start'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:172:in `transmit'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:67:in `post'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.0.14/lib/apipie_bindings/api.rb:286:in `call_client'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.0.14/lib/apipie_bindings/api.rb:217:in `http_call'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.0.14/lib/apipie_bindings/api.rb:162:in `call'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.0.14/lib/apipie_bindings/resource.rb:14:in `call'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.5.1.3/lib/hammer_cli/apipie/command.rb:43:in `send_request'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_foreman-0.5.1.2/lib/hammer_cli_foreman/commands.rb:189:in `send_request'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_foreman_tasks-0.0.10/lib/hammer_cli_foreman_tasks/async_command.rb:14:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/command.rb:68:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.5.1.3/lib/hammer_cli/abstract.rb:22:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/subcommand/execution.rb:11:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/command.rb:68:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.5.1.3/lib/hammer_cli/abstract.rb:22:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/subcommand/execution.rb:11:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/command.rb:68:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.5.1.3/lib/hammer_cli/abstract.rb:22:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/command.rb:133:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.5.1.3/bin/hammer:125:in `<top (required)>'
    /usr/bin/hammer:23:in `load'
    /usr/bin/hammer:23:in `<main>'
Comment 7 Og Maciel 2016-03-18 16:30 EDT
Created attachment 1137866 [details]
Settings for exporting

It is necessary to first define a path (writable by the apache user) for the 'pulp_export_destination' setting, as shown here.
Comment 8 Og Maciel 2016-03-18 16:48:28 EDT
To verify this issue:

* Update your 'pulp_export_destination' setting and set it to "/var/www/html/pub" (web UI, Settings menu, search for pulp_export_destination)
* Create a repository (I used an existing, synchronized RHEL repo) and sync it
* Use the hammer repository export command as per first comment here
Comment 9 Og Maciel 2016-03-18 16:49 EDT
Created attachment 1137881 [details]
Stacktrace generated when exporting a repository via hammer
Comment 10 Og Maciel 2016-03-18 16:50:15 EDT
Mike McCune also could not test this feature, so I am failing it.

Tested against Satellite 6.2.0 SNAP 4.0 build
Comment 11 Chris Duryee 2016-03-21 14:14:28 EDT
There are a few issues found:

1. there was a missing cherry-pick for katello-selinux which will be in the next snap. I ran "audit2allow -a" and saw "allow passenger_t httpd_sys_rw_content_t:dir search;". After updating the selinux policy, you should see a message like "#!!!! This avc is allowed in the current policy" which indicates that selinux is copacetic. Note that fresh installs that didnt have a prior denial here will not have any message, it will just work:)

2. the directory needs to be owned by foreman user and group, not apache. This is documented in the upstream docs at http://www.katello.org/docs//user_guide/disconnected/, downstream docs are being updated for this (https://bugzilla.redhat.com/show_bug.cgi?id=1285244#c50 and https://bugzilla.redhat.com/show_bug.cgi?id=1285244#c55) I think it was apache.apache in the past, probably on one of the etherpads at one point.

3. default selinux policy does not allow passenger to write to /var/www/html/*, even if the file-level permissions are correct.

Typically I export repos to /mnt/exports, so I can pretend that I'm exporting to a mount that can be shared. I don't know how common it will be to export to /var/www/html/pub for the disconnected use case, since it short-circuits the "export, walk it over, then import" flow.

Having said that, we can add an additional selinux rule to allow issue 3 if you think it would be helpful. It would be impossible to guess all the places someone would export and create rules in advance, so maybe a kbase article would be more helpful that shows how to do it generically.

Marking bz as NEEDSINFO on omaciel for to get feedback on third issue.
Comment 12 Chris Duryee 2016-03-21 14:25:26 EDT
Something I should have mentioned, for issue 3 in comment #11 the selinux denial only affects certain directories like /var/www/html/. Dirs like /mnt/export should be unaffected and do not require additional rules.
Comment 13 Og Maciel 2016-03-21 14:36:16 EDT
Hi Chris,

[root@ibm-x3250m4-01 pub]# mkdir /mnt/export
[root@ibm-x3250m4-01 pub]# chown foreman.foreman /mnt/export
[root@ibm-x3250m4-01 pub]# ls -l /mnt/export/
total 0
[root@ibm-x3250m4-01 pub]# ls -ld /mnt/export/
drwxr-xr-x. 2 foreman foreman 6 Mar 21 14:34 /mnt/export/
[root@ibm-x3250m4-01 pub]# LANG=en_US.UTF-8 hammer -v -u admin -p changeme  repository export --id 1
[ERROR 2016-03-21 14:35:06 Exception] ERF42-6337 [Foreman::Exception]: Unable to export, 'pulp_export_destination' setting is not a writable directory.
Could not export the repository:
  ERF42-6337 [Foreman::Exception]: Unable to export, 'pulp_export_destination' setting is not a writable directory.
[ERROR 2016-03-21 14:35:06 Exception]

RestClient::InternalServerError (500 Internal Server Error):
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:230:in `process_result'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:178:in `block in transmit'
    /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:853:in `start'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:172:in `transmit'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:67:in `post'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.0.14/lib/apipie_bindings/api.rb:286:in `call_client'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.0.14/lib/apipie_bindings/api.rb:217:in `http_call'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.0.14/lib/apipie_bindings/api.rb:162:in `call'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.0.14/lib/apipie_bindings/resource.rb:14:in `call'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.5.1.3/lib/hammer_cli/apipie/command.rb:43:in `send_request'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_foreman-0.5.1.2/lib/hammer_cli_foreman/commands.rb:189:in `send_request'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_foreman_tasks-0.0.10/lib/hammer_cli_foreman_tasks/async_command.rb:14:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/command.rb:68:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.5.1.3/lib/hammer_cli/abstract.rb:22:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/subcommand/execution.rb:11:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/command.rb:68:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.5.1.3/lib/hammer_cli/abstract.rb:22:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/subcommand/execution.rb:11:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/command.rb:68:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.5.1.3/lib/hammer_cli/abstract.rb:22:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.0.0/lib/clamp/command.rb:133:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.5.1.3/bin/hammer:125:in `<top (required)>'
    /usr/bin/hammer:23:in `load'
    /usr/bin/hammer:23:in `<main>'

By the way, I think that a KB article would be great for those who like myself are struggling a bit :)
Comment 16 Chris Duryee 2016-03-28 10:24:17 EDT
Og,

I created https://bugzilla.redhat.com/show_bug.cgi?id=1321589 so you don't have to do the steps noted in issue 3. Jitendra hit it as well, so IMO it is a common enough use case to warrant adding a rule.
Comment 18 Og Maciel 2016-04-04 11:37:55 EDT
QE: I have been able to test this issue using a small YUM repo but had issues when exporting a large-ish Red Hat repository. To properly verify this issue I think we need to:

* Test exporting Background download policy repos
* Test exporting OnDemand download policy repos
* Verify https://bugzilla.redhat.com/show_bug.cgi?id=1323730
Comment 24 Lukas Pramuk 2016-04-25 09:47:19 EDT
New BZ to raise a correct error message when exporting individual non-yum repos
 BZ #1330166
Comment 26 errata-xmlrpc 2016-07-27 05:24:05 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501

Note You need to log in before you can comment on or make changes to this bug.