Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1504635

Summary: Stop calling manifest export when doing manifest refresh
Product: Red Hat Satellite Reporter: Pavel Moravec <pmoravec>
Component: Subscription ManagementAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Peter Ondrejka <pondrejk>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.2.12CC: ajoseph, bbuckingham, ehelms, inecas, janarula, jsherril, mwade, pcreech, pmoravec, smeyer, wlehman, wpinheir
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-21 16:54:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pavel Moravec 2017-10-20 10:50:58 UTC
Description of problem:
Using one particular manifest from a customer, an attempt to refresh it fails with backtrace:

 Error:

RestClient::BadRequest

400 Bad Request

---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in
  `return!'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:231:in
  `process_result'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.38.1/lib/restclient_ext/request.rb:52:in
  `block in transmit'"
- "/opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:853:in `start'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.38.1/lib/restclient_ext/request.rb:44:in
  `transmit'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:51:in
  `get'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.154/app/lib/katello/resources/candlepin.rb:291:in
  `export'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.154/app/models/katello/glue/provider.rb:179:in
  `owner_upstream_export'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.154/app/models/katello/glue.rb:164:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.154/app/models/katello/glue.rb:114:in
  `block in process'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.154/app/models/katello/glue.rb:101:in
  `each'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.154/app/models/katello/glue.rb:101:in
  `process'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.154/app/models/katello/glue.rb:23:in
  `on_save'"
..


It happens when calling:

2017-10-20 12:13:47  [katello/cp_rest] [D] Sending GET request to upstream Candlepin: https://subscription.rhn.redhat.com/subscription/consumers/fe7b307e-3649-470b-887d-53fc434ff72e/export

that is responded by a minute silence from subscription.rhn.redhat.com, followed by timeout from Satellite side and connection termination.

Anyway, mimicking the same call from cmdline, it succeeds:

# time wget --no-check-certificate --certificate=CERT.pem --private-key=KEY.pem https://subscription.rhn.redhat.com/subscription/consumers/fe7b307e-3649-470b-887d-53fc434ff72e/export
--2017-10-20 11:05:58--  https://subscription.rhn.redhat.com/subscription/consumers/fe7b307e-3649-470b-887d-53fc434ff72e/export
Resolving subscription.rhn.redhat.com (subscription.rhn.redhat.com)... 10.4.204.34
Connecting to subscription.rhn.redhat.com (subscription.rhn.redhat.com)|10.4.204.34|:443... connected.
WARNING: cannot verify subscription.rhn.redhat.com's certificate, issued by ‘/C=US/ST=North Carolina/O=Red Hat, Inc./OU=Red Hat Network/CN=Red Hat Entitlement Operations Authority/emailAddress=ca-support’:
  Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: ‘export.1’

    [                                <=>                                                                                                                  ] 6,425,704    783KB/s   in 8.3s   

2017-10-20 11:06:13 (761 KB/s) - ‘export.1’ saved [6425704]


real	0m14.471s
user	0m0.033s
sys	0m0.053s

#


So the problem isnt on subscription.rhn.redhat.com but in Satellite.


Problem is reproducible with the given manifest (will attach it in a private comment) on any Satellite we tried. Nothing weird found in the way how gems rest-client invokes the http request.


Version-Release number of selected component (if applicable):
Sat 6.2.12


How reproducible:
100%


Steps to Reproduce:
1. Import given manifest (attachment will follow)
2. Try to refresh it


Actual results:
Refresh fails with above error


Expected results:
No such error.


Additional info:
From tcpdump: 
- Satellite-based request was sent, no response received in 1 minute, then Satellite gave up.
- For wget-based request, redhat.com started to respond 5s after the request and whole wget finished within 15s. So [1] isnt supposed to help (but will try).

[1] https://github.com/Katello/katello/pull/6871

Comment 2 Pavel Moravec 2017-10-20 11:43:24 UTC
OK, the [1] (candlepin response timeout increase) _is_ resolution. In particular changes per [2].

[1] https://github.com/Katello/katello/pull/6871
[2] https://github.com/Katello/katello/pull/6871/files#diff-d4ada19367f2e37f2adeafa9de778bcc

Comment 3 Satellite Program 2017-10-20 12:12:39 UTC
Upstream bug assigned to jsherril

Comment 4 Satellite Program 2017-10-20 12:12:41 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/20308 has been resolved.

Comment 5 Stefan Meyer 2017-10-20 12:13:18 UTC
In katello-3.0.0.157 this fails after 5 minutes. So the 2 line fix

RestClient::Resource.new(url,
   :ssl_client_cert => OpenSSL::X509::Certificate.new(client_cert),
   :ssl_client_key => OpenSSL::PKey::RSA.new(client_key),
   :ssl_ca_file => ca_file,
   :verify_ssl => ca_file ? OpenSSL::SSL::VERIFY_PEER : OpenSSL::SSL::VERIFY_NONE,
   :open_timeout => 60*20,   ### added
   :timeout => 60*20         ### added
)

doesn't seem to work. It just pushes the time to fail from around 1 minute to 5 minutes.

Comment 6 Pavel Moravec 2017-10-20 12:30:41 UTC
Per bcourt, since lst weekend these calls are not further required. So then this should be commented out (together with the owner_upstream_export method being called just from here)?

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.154/app/models/katello/glue/provider.rb :

         if manifest_refresh
            zip_file_path = "/tmp/#{rand}.zip"
            upstream = options[:upstream]
            pre_queue.create(:name => "regenerate upstream certificates for: #{self.organization.name}",
                             :priority => 1, :action => [self, :owner_regenerate_upstream_certificates, upstream],
                             :action_rollback => nil)
            pre_queue.create(:name => "export upstream manifest for owner: #{self.organization.name}",
                             :priority => 2, :action => [self, :owner_upstream_update, upstream, options],
                             :action_rollback => nil)
#            pre_queue.create(:name => "export upstream manifest for owner: #{self.organization.name}",
#                             :priority => 3, :action => [self, :owner_upstream_export, upstream, zip_file_path, options],
#                             :action_rollback => nil)
          else

Comment 7 Justin Sherrill 2017-10-20 12:38:07 UTC
Stephan, 

60*20 is 20 minutes, so i'm not sure why its not working.

Pavel, yes now that candlepin 2.0 is in production, this isn't required at all, and those lines can be commented out.

Comment 8 Pavel Moravec 2017-10-20 16:41:19 UTC
Thanks Justin.

Let have this bug as improvement to stop calling the manifest export from subscription.rhn.redhat.com, i.e. implement the https://bugzilla.redhat.com/show_bug.cgi?id=1504635#c6 .

Comment 14 Ivan Necas 2017-11-24 08:56:29 UTC
Pavel, Justin: I'm don't understand how https://bugzilla.redhat.com/show_bug.cgi?id=1504635#c6 should work with commenting out calling the export method: as we use the path in the other parts of the orchestration.

Comment 15 Pavel Moravec 2017-11-24 10:40:58 UTC
(In reply to Ivan Necas from comment #14)
> Pavel, Justin: I'm don't understand how
> https://bugzilla.redhat.com/show_bug.cgi?id=1504635#c6 should work with
> commenting out calling the export method: as we use the path in the other
> parts of the orchestration.

For the particular call flow, the commented-out code helps. If that code is really used by other call flows (I dont know if so or not), then it cant be used as a "permanent" solution but only as a workaround "comment out, refresh manifest, uncomment".

Comment 18 Stefan Meyer 2017-11-24 14:12:33 UTC
To clarify, for everyone looking for a hotfix.

You need to implement https://bugzilla.redhat.com/show_bug.cgi?id=1504635#c5 first.

https://bugzilla.redhat.com/show_bug.cgi?id=1504635#c6 is the wrong configuration. The code should look like this:

         if manifest_refresh
            zip_file_path = "/tmp/#{rand}.zip"
            upstream = options[:upstream]
            #pre_queue.create(:name => "regenerate upstream certificates for: #{self.organization.name}",
            #                 :priority => 1, :action => [self, :owner_regenerate_upstream_certificates, upstream],
            #                 :action_rollback => nil)
            pre_queue.create(:name => "export upstream manifest for owner: #{self.organization.name}",
                             :priority => 2, :action => [self, :owner_upstream_update, upstream, options],
                             :action_rollback => nil)
            pre_queue.create(:name => "export upstream manifest for owner: #{self.organization.name}",
                             :priority => 3, :action => [self, :owner_upstream_export, upstream, zip_file_path, options],
                             :action_rollback => nil)
          else
            zip_file_path = options[:zip_file_path]
          end

Depending on the Satellite version this must be in /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.XXX/app/models/katello/glue/provider.rb

Comment 20 Satellite Program 2017-11-27 19:11:47 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/21493 has been resolved.

Comment 23 Peter Ondrejka 2017-12-04 13:41:48 UTC
Verified on Satellite 6.3 snap 27 using the provided manifest, manifest refresh completes successfully.

Comment 24 Justin Sherrill 2017-12-18 15:34:36 UTC
Clearing needinfo

Comment 27 Satellite Program 2018-02-21 16:54:17 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> 
> For information on the advisory, and where to find the updated files, follow the link below.
> 
> If the solution does not work for you, open a new bug report.
> 
> https://access.redhat.com/errata/RHSA-2018:0336