Bug 1338632

Summary: Syncing ostree contents from CDN raises error: [E] OST0005: Fetch summary failed. Reason: GLib.Error
Product: Red Hat Satellite Reporter: Sachin Ghai <sghai>
Component: Content ManagementAssignee: Partha Aji <paji>
Status: CLOSED ERRATA QA Contact: Sachin Ghai <sghai>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2.0CC: bbuckingham, ehelms, jomitsch, jortel, mmccune, sghai, xdmoon
Target Milestone: UnspecifiedKeywords: Regression, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libtasn1-3.8-3 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-27 11:34:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ostree sync is throwing eason: GLib.Error('Error creating directory: File exists', 'g-io-error-quark', 2)
none
complete traceback
none
sync completed for ostree contents none

Description Sachin Ghai 2016-05-23 07:35:58 UTC
Description of problem:
I installed a setup with sat6.2 GA snap11 and synced Atomic ostree from CDN. Later, I upgraded that setup with latest snap12.1. Upgrade was successful.

But when I re-sync ostree repo from CDN: Sync failed with error:


2016-05-23 03:29:06 [app] [I] Started POST "/katello/sync_management/sync" for 10.65.223.218 at 2016-05-23 03:29:06 -0400
2016-05-23 03:29:06 [app] [I] Processing by Katello::SyncManagementController#sync as JS
2016-05-23 03:29:06 [app] [I]   Parameters: {"utf8"=>"✓", "sync_status_url"=>"", "repoids"=>["7"], "commit"=>"Synchronize Now"}
2016-05-23 03:29:06 [app] [I] Completed 200 OK in 613ms (Views: 0.3ms | ActiveRecord: 53.3ms)
2016-05-23 03:29:07 [app] [I] Started GET "/katello/sync_management/sync_status?repoids%5B%5D=7&_=1463988484024" for 10.65.223.218 at 2016-05-23 03:29:07 -0400
2016-05-23 03:29:07 [app] [I] Processing by Katello::SyncManagementController#sync_status as JSON
2016-05-23 03:29:07 [app] [I]   Parameters: {"repoids"=>["7"], "_"=>"1463988484024"}
2016-05-23 03:29:07 [app] [I] Completed 200 OK in 41ms (Views: 0.1ms | ActiveRecord: 4.6ms)
2016-05-23 03:29:08 [foreman-tasks/action] [E] OST0005: Fetch summary failed. Reason: GLib.Error('Server returned status 403: Forbidden', 'g-io-error-quark', 0) (Katello::Errors::PulpError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.32/app/lib/actions/pulp/abstract_async_task.rb:121:in `block in external_task='
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.32/app/lib/actions/pulp/abstract_async_task.rb:119:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.32/app/lib/actions/pulp/abstract_async_task.rb:119:in `external_task='
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.32/app/lib/actions/pulp/repository/sync.rb:46:in `external_task='
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/polling.rb:98:in `poll_external_task_with_rescue'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/polling.rb:21:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/cancellable.rb:9:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.32/app/lib/actions/pulp/abstract_async_task.rb:45:in `run'


I think its a regression, as sync was working before with 6.2 GA snap11.

Version-Release number of selected component (if applicable):
6.2 GA snap12.1

How reproducible:
always

Steps to Reproduce:
1. sync "Red Hat Enterprise Linux Atomic Host Trees" from CDN
2.
3.

Actual results:
sync incomplete: error: [E] OST0005: Fetch summary failed. Reason: GLib.Error('Server returned status 403: Forbidden', 'g-io-error-quark', 0) (Katello::Errors::PulpError)

Expected results:
sync should work

Additional info:

Comment 3 John Mitsch 2016-05-23 13:02:19 UTC
This is possibly related to this bug and the discussion here: https://bugzilla.redhat.com/show_bug.cgi?id=1323644

Comment 4 Sachin Ghai 2016-05-23 13:40:46 UTC
On a fresh installed sat6.2 snap12.1, ostree sync never finishes, its keep on running from last 38 minutes.. So I filed this bz. https://bugzilla.redhat.com/show_bug.cgi?id=1338632

Comment 5 Brad Buckingham 2016-05-23 16:59:37 UTC
John,  I am going to triage this one to you.  If the cause is the same as bug 1323644, please close this one as a duplicate.

Comment 6 John Mitsch 2016-05-23 18:23:54 UTC
Sachin,

Partha and I had a hard time reproducing this bug. Can you make sure that you are still able to reproduce it?

Comment 7 Sachin Ghai 2016-05-30 09:44:16 UTC
To reproduce the reported issue I just updated the setup from sanp12.1 to snap13.1

and re-sync the atomic repo and this time I see below issue:

2016-05-30 05:39:58 [foreman-tasks/action] [E] OST0001: Create local repository at: /var/lib/pulp/content/shared/ostree/f3769796a028aff68cf0e680a3e84bd236833ad48c2700f1e593ea584e5e6abf/content failed. Reason: GLib.Error('Error creating directory: File exists', 'g-io-error-quark', 2) (Katello::Errors::PulpError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.35/app/lib/actions/pulp/abstract_async_task.rb:121:in `block in external_task='
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.35/app/lib/actions/pulp/abstract_async_task.rb:119:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.35/app/lib/actions/pulp/abstract_async_task.rb:119:in `external_task='
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.35/app/lib/actions/pulp/repository/sync.rb:48:in `external_task='
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/polling.rb:98:in `poll_external_task_with_rescue'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/polling.rb:21:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/cancellable.rb:9:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.35/app/lib/actions/pulp/abstract_async_task.rb:45:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:506:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'


Since this execption is again related to GLib Error so updated in same bz. But if you want me to file separate issue, just let me know.

Comment 8 Sachin Ghai 2016-05-30 09:46:54 UTC
Created attachment 1162751 [details]
ostree sync is throwing eason: GLib.Error('Error creating directory: File exists', 'g-io-error-quark', 2)

Comment 9 John Mitsch 2016-06-07 19:02:42 UTC
Jeff,

Do you have any idea about this bug and these errors? I am thinking this may be related to https://bugzilla.redhat.com/show_bug.cgi?id=1329051

Comment 10 Jeff Ortel 2016-06-07 19:18:12 UTC
Hard to say without seeing the entire trace.  Sachin, do you have multiple ostree repositories with the same feed URL that could have been sync'ing at the same time?

Comment 11 Sachin Ghai 2016-06-13 11:01:07 UTC
@Jeff: Nop.. I do have only one ostree repository enabled ("Red Hat Enterprise Linux Atomic Host Trees")

Comment 12 Sachin Ghai 2016-06-13 11:01:39 UTC
Created attachment 1167426 [details]
complete traceback

Comment 14 Sachin Ghai 2016-06-13 11:03:50 UTC
Please note that on fresh install, the reported issue is no longer reproducible because we are hitting another issue: https://bugzilla.redhat.com/show_bug.cgi?id=1329051

Comment 15 Partha Aji 2016-07-13 15:48:51 UTC
This should now work with updates from https://bugzilla.redhat.com/show_bug.cgi?id=1330753

Comment 16 Sachin Ghai 2016-07-18 10:40:11 UTC
Verified with sat6.1.9 -> sat62 GA snap20.2

I can see sync ostree contents from CDN in approx 7 minutes and can re-sync same contents after upgrade from sanp19 -20.2 in approx (2 minutes). Please see attached screeshot for verification.

Comment 17 Sachin Ghai 2016-07-18 10:58:20 UTC
Created attachment 1181020 [details]
sync completed for ostree contents

Comment 18 Bryan Kearney 2016-07-27 11:34:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501