Bug 1466482 - RHEL Atomic Tree (Version 7.3.5-1) sync fails
RHEL Atomic Tree (Version 7.3.5-1) sync fails
Status: NEW
Product: Red Hat Update Infrastructure for Cloud Providers
Classification: Red Hat
Component: RHUA (Show other bugs)
3.0.0
Unspecified Unspecified
high Severity unspecified
: ---
: ---
Assigned To: RHUI Bug List
Vratislav Hutsky
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-29 14:30 EDT by Andrew Blum
Modified: 2017-08-04 15:32 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
sosreport from rhua after failed sync (8.07 MB, application/x-xz)
2017-06-29 14:30 EDT, Andrew Blum
no flags Details

  None (edit)
Description Andrew Blum 2017-06-29 14:30:54 EDT
Created attachment 1292951 [details]
sosreport from rhua after failed sync

Description of problem:
Synchronization of RHEL Atomic Host (Trees) from RHUI fails for (Version 7.3.5-1) with exception "Pulling remote refs failed. Reason: GLib.Error('Download incomplete', 'g-io-error-quark', 0)"

Version-Release number of selected component (if applicable):
Fresh install of RHUI 3.0.1 for RHEL 7.  Environment includes 1 RHUA, use of glusterfs storage provided by a single CDS node, and 1 HAPROXY server.  Installation steps based on: https://access.redhat.com/documentation/en-us/red_hat_update_infrastructure/3.0/html/system_administrators_guide/registration_and_subscriptions
 

How reproducible:
Fails every time for that repo (though was successful with the older 7.3.4-1 tree), but other repos can be successfully sync'd like:

Next Sync                    Last Sync                    Last Result         
------------------------------------------------------------------------------
Red Hat Enterprise Linux 7 Server - Extras from RHUI (RPMs) (x86_64)
06-29-2017 16:54             06-29-2017 10:57             Success   

Steps to Reproduce:
1. Use rhui-manager to add repository "Red Hat Enterprise Linux Atomic Host (Trees) from RHUI (Version 7.3.5-1)"
2. Initiate synchronization via rhui-manager
3. Wait about 10 minutes and the following is displayed:

Next Sync                    Last Sync                    Last Result         
------------------------------------------------------------------------------
Red Hat Enterprise Linux Atomic Host (Trees) from RHUI (Version 7.3.5-1)
06-29-2017 17:46             06-29-2017 11:52             Error     


Expected results:
Synchronization should result in "Success".


Additional info:
The traceback provided:

  Start Time:     06-29-2017 11:47
  Finish Time:    06-29-2017 11:52
  Elapsed Time:   0:04:58
  Result:         Success
  Traceback:      Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__
    return super(Task, self).__call__(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
    return super(PulpTask, self).__call__(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 760, in sync
    sync_report = sync_repo(transfer_repo, conduit, call_config)
  File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 658, in wrap_f
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers/web.py", line 91, in sync_repo
    report = step.process_lifecycle()
  File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 562, in process_lifecycle
    super(PluginStep, self).process_lifecycle()
  File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 159, in process_lifecycle
    step.process()
  File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 249, in process
    self._process_block()
  File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 293, in _process_block
    self.process_main()
  File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers/steps.py", line 159, in process_main
    self._pull(self.parent.storage_dir, self.parent.repo_id, self.parent.branches)
  File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers/steps.py", line 187, in _pull
    raise pe
PulpCodedException: Pulling remote refs failed. Reason: GLib.Error('Download incomplete', 'g-io-error-quark', 0)


NOTE:: This environment can be made available again for additional troubleshooting as needed.  It is currently saved as a "blueprint" in a cloud-based environment provided by Ravello/Oracle (ravellosystems.com).
Comment 3 Radek Bíba 2017-07-03 03:22:59 EDT
This might be a variation on bug 1427190, which is also mentioned in https://access.redhat.com/documentation/en-us/red_hat_update_infrastructure/3.0/html/release_notes/known_issues (the last item in the list). Have you tried syncing the repo again?
Comment 4 Karan Rai 2017-07-05 06:27:28 EDT
In the same environment, with the newer version -
====
Red Hat Enterprise Linux Atomic Host (Trees) from RHUI (Version 7.3.6)
====

Sync is still failing, also tried syncing multiple times.
-------------------
[root@rhua ~]# pulp-admin -u admin -p admin tasks details --task-id 39e46a51-e6ba-46cc-bed4-e933991ef879
+----------------------------------------------------------------------+
                              Task Details
+----------------------------------------------------------------------+

Operations:       sync
Resources:        rhel-atomic-host-rhui-ostree--rhel-atomic-host-7-x86_64-standa
                  rd (repository)
State:            Failed
Start Time:       2017-07-05T08:04:15Z
Finish Time:      2017-07-05T08:11:44Z
Result:           N/A
Task Id:          39e46a51-e6ba-46cc-bed4-e933991ef879
Progress Report:  
  Ostree Web Importer: 
    Description:   Create Local Repository
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 1
    Num Success:   1
    State:         FINISHED
    Step Id:       c2ddc887-f050-41d8-aaad-667f52a64bec
    Step Type:     import_create_repository
    Description:   Update Summary
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 1
    Num Success:   1
    State:         FINISHED
    Step Id:       53e3ae25-6084-4dbf-aead-380be3d7f629
    Step Type:     import_summary
    Description:   Pull Remote Branches
    Details:       fetching 2980/17850 16%
    Error Details: 
      Error:     Pulling remote refs failed. Reason: GLib.Error('Download
                 incomplete', 'g-io-error-quark', 0)
      Traceback:   File
                 "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_ste
                 p.py", line 249, in process     self._process_block()    File
                 "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_ste
                 p.py", line 293, in _process_block     self.process_main()   
                 File
                 "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers
                 /steps.py", line 159, in process_main    
                 self._pull(self.parent.storage_dir, self.parent.repo_id,
                 self.parent.branches)    File
                 "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers
                 /steps.py", line 187, in _pull     raise pe 
    Items Total:   1
    Num Failures:  1
    Num Processed: 1
    Num Success:   0
    State:         FAILED
    Step Id:       5796cdac-7cd1-4848-a7cb-e79c58380dca
    Step Type:     import_pull
    Description:   Add Content Units
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 0
    Num Success:   0
    State:         NOT_STARTED
    Step Id:       b15f6b70-1430-4585-98d2-9e525ca49344
    Step Type:     import_add_unit
    Description:   Clean
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 0
    Num Success:   0
    State:         NOT_STARTED
    Step Id:       5bb8b842-f73a-4879-b033-e16283dccfb2
    Step Type:     import_clean
Traceback:        Traceback (most recent call last):   File
                  "/usr/lib/python2.7/site-packages/celery/app/trace.py", line
                  240, in trace_task     R = retval = fun(*args, **kwargs)  
                  File
                  "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py",
                  line 473, in __call__     return super(Task,
                  self).__call__(*args, **kwargs)   File
                  "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py",
                  line 103, in __call__     return super(PulpTask,
                  self).__call__(*args, **kwargs)   File
                  "/usr/lib/python2.7/site-packages/celery/app/trace.py", line
                  437, in __protected_call__     return self.run(*args,
                  **kwargs)   File
                  "/usr/lib/python2.7/site-packages/pulp/server/controllers/repo
                  sitory.py", line 760, in sync     sync_report =
                  sync_repo(transfer_repo, conduit, call_config)   File
                  "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py",
                  line 658, in wrap_f     return f(*args, **kwargs)   File
                  "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importer
                  s/web.py", line 91, in sync_repo     report =
                  step.process_lifecycle()   File
                  "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_st
                  ep.py", line 562, in process_lifecycle     super(PluginStep,
                  self).process_lifecycle()   File
                  "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_st
                  ep.py", line 159, in process_lifecycle     step.process()  
                  File
                  "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_st
                  ep.py", line 249, in process     self._process_block()   File
                  "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_st
                  ep.py", line 293, in _process_block     self.process_main()  
                  File
                  "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importer
                  s/steps.py", line 159, in process_main    
                  self._pull(self.parent.storage_dir, self.parent.repo_id,
                  self.parent.branches)   File
                  "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importer
                  s/steps.py", line 187, in _pull     raise pe
                  PulpCodedException: Pulling remote refs failed. Reason:
                  GLib.Error('Download incomplete', 'g-io-error-quark', 0) 
-------------------
Comment 5 Radek Bíba 2017-07-11 08:14:02 EDT
Ah, so it must be something else. The sync task works for me, though; here's what I can see in the output from "pulp-admin tasks list --all":

Operations:  sync
Resources:   rhel-atomic-host-rhui-ostree--rhel-atomic-host-7-x86_64-standard
             (repository)
State:       Successful
Start Time:  2017-07-11T11:11:49Z
Finish Time: 2017-07-11T11:29:48Z
Task Id:     84142918-15b4-4bce-a051-2b37e7088707

Operations:  publish
Resources:   rhel-atomic-host-rhui-ostree--rhel-atomic-host-7-x86_64-standard
             (repository)
State:       Successful
Start Time:  2017-07-11T11:29:48Z
Finish Time: 2017-07-11T11:29:48Z
Task Id:     5eaa4ea0-f7a0-45bf-87ea-4873ffade103

"pulp-admin tasks details --task-id 84142918-15b4-4bce-a051-2b37e7088707" also looks good on my test machine.

Could it be a network problem? Note that the Atomic repo consists of ~17k files of various (but generally small) sizes, so perhaps a network node has a hard time handling all the fragmented traffic? Otherwise, I can't think of a good reason for this kind of issue.

Note You need to log in before you can comment on or make changes to this bug.