Bug 1369892 - OSTree Sync pulls only the latest version
Summary: OSTree Sync pulls only the latest version
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: 6.4.0
Assignee: Partha Aji
QA Contact: Kedar Bidarkar
URL:
Whiteboard:
: 1514009 (view as bug list)
Depends On: 1425605
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-24 15:51 UTC by Partha Aji
Modified: 2021-12-10 14:43 UTC (History)
22 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-16 19:21:15 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Failed: Satellite 6.3: Ostree branches only shows 1 (56.76 KB, image/png)
2017-08-30 11:46 UTC, Brad Buckingham
no flags Details
OSTree sync showing all branches of RHEL7 (105.14 KB, image/png)
2018-07-06 09:48 UTC, Kedar Bidarkar
no flags Details
OSTree branches sync Coomplete (64.13 KB, image/png)
2018-07-06 09:49 UTC, Kedar Bidarkar
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 17427 0 Normal Closed OSTree Sync pulls only the latest version 2020-10-14 03:35:41 UTC
Foreman Issue Tracker 18076 0 Normal Closed ostree 'Depth' parameter support needed in runcible 2020-10-14 03:35:41 UTC
Pulp Redmine 2205 0 High CLOSED - CURRENTRELEASE As a user, I want to pull older versions of trees available to atomic hosts. 2016-12-19 16:01:44 UTC
Pulp Redmine 3210 0 Normal CLOSED - CURRENTRELEASE Update Importer configuration fails. 2018-01-30 19:31:43 UTC
Red Hat Knowledge Base (Solution) 3234861 0 None None None 2017-11-14 01:17:41 UTC

Description Partha Aji 2016-08-24 15:51:33 UTC
As of 6.2 OSTree sync only pulls the latest branch, it does not sync the full history. This affects customers wanting to deploy to a specific version of the host.

Steps:
1) Sat with ostree enabled
2) manifest with ostree + enable atomic repo or create product + repo with one of the upstream repos.
3) Sync
Actual 
Notice only the latest version (7.2.6) as of now shows up.

Expected
All versions traversable via history,

Comment 1 Jeff Ortel 2016-08-24 16:59:52 UTC
What do we want the default depth to be?

Do we need to included the additional commits (and metadata) in the summary information pulp stores in the (pulp) repository scratchpad?

Comment 3 Partha Aji 2016-08-30 15:25:29 UTC
I would say lets keeps the depth infinite and get full history by default. The user using this is likely going to want to preserve history and use the ability to roll back. We can always override specific repos if history is not desired.

We are not using scratch pad from summary in Katello. We assume pulp does a pull all and populates its mongo db accordingly. We use :type_ids => ["ostree"] to get the branch information from there.

Comment 4 pulp-infra@redhat.com 2016-08-30 15:30:29 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 5 pulp-infra@redhat.com 2016-08-30 15:30:33 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 6 Jeff Ortel 2016-08-30 21:42:41 UTC
Note: When testing with F24 trees at: https://kojipkgs.fedoraproject.org/atomic/24/ using a depth=-1, the sync time slightly more than doubled.  Went from 7 min (depth=0) and 16 min (depth=-1).

Only pulled (1) branch.

Comment 7 pulp-infra@redhat.com 2016-08-30 22:00:26 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 8 Jeff Ortel 2016-09-14 15:17:42 UTC
This commit will need to be cherry picked downstream.

Comment 9 pulp-infra@redhat.com 2016-09-14 15:31:40 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 10 Jeff Ortel 2016-09-14 15:33:23 UTC
Here is the commit: 7735b5c8ba2e5b9bfeb4d6228b5956cd3402e6c9

Comment 11 Jeff Ortel 2016-09-14 15:38:11 UTC
Katello will need to configure both the importer and distributor with depth=-1 for this to be completely resolved.

Comment 15 pulp-infra@redhat.com 2016-10-26 05:01:32 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 16 Brad Buckingham 2016-11-21 18:17:49 UTC
Created redmine issue http://projects.theforeman.org/issues/17427 from this bug

Comment 17 Bryan Kearney 2016-12-16 19:11:03 UTC
Upstream bug component is Repositories

Comment 18 pulp-infra@redhat.com 2016-12-19 16:01:45 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 19 Satellite Program 2017-04-06 16:11:44 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/18076 has been resolved.

Comment 20 John Mitsch 2017-08-02 20:00:04 UTC
verification blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1477751

Comment 21 Brad Buckingham 2017-08-30 11:45:30 UTC
Failed QA with Satellite 6.3 SNAP 13: satellite-6.3.0-16.0.beta.el7sat.noarch , tfm-rubygem-katello-3.4.4-1.el7sat.noarch


- imported manifest
- enabled : Red Hat Enterprise Linux Atomic Host Trees
- synced : Red Hat Enterprise Linux Atomic Host Trees

- after syncing, there is only 1 branch visible
- attaching screenshot of the result

Comment 22 Brad Buckingham 2017-08-30 11:46:08 UTC
Created attachment 1320039 [details]
Failed: Satellite 6.3: Ostree branches only shows 1

Comment 23 pulp-infra@redhat.com 2017-08-30 12:03:25 UTC
Requesting needsinfo from upstream developer jortel because the 'FailedQA' flag is set.

Comment 24 Tanya Tereshchenko 2017-08-30 15:06:39 UTC
Could you post details of the API calls which Katello does in this scenario?
To be sure that depth=-1 is configured for both importer and distributor.

Comment 25 Jeff Ortel 2017-08-30 15:23:02 UTC
Tanya is asking the appropriate question.

Comment 26 pulp-infra@redhat.com 2017-08-30 15:33:28 UTC
Requesting needsinfo from upstream developer jortel because the 'FailedQA' flag is set.

Comment 27 Brad Buckingham 2017-08-30 16:20:52 UTC
When initially testing this, it was not obvious from the description how depth actually works within Satellite.

After some digging around, I now see the following as a configurable attribute on the repository:
   attribute: "Upstream Sync Policy"
   possible values: "Latest", "All History" and "Custom Depth".

In the initial test, it was set to the default behavior, "Latest", which appears to generate the correct behavior.  Only 1 branch is synced.

Setting it to "Custom Depth" of 3 and sync, the result was 2 branches that appear to be identical from 'Content => OSTree Branches'.  Is that expected behavior?

Setting it to "All History" and sync, the result was 3 branches that appear to be  identical from 'Content => OSTree Branches'.  Is that expected behavior?

Comment 28 pulp-infra@redhat.com 2017-08-30 16:35:05 UTC
Requesting needsinfo from upstream developer jortel because the 'FailedQA' flag is set.

Comment 29 Partha Aji 2017-09-19 17:05:50 UTC
Hey Brad,
For detailed description and verification steps look at https://github.com/Katello/katello/pull/6637#issuecomment-288629620

I suspect you ran into the

"""
Note that the UI metrics of a sync operation are going to look identical whether you are syncing using All History/Latest Only/Or Custom even though these mean different traversal depth. This is because pulp does not capture "all" the results of the sync in its mongo db. It only captures the "branch heads" and makes units out of them. So Katello has no way of knowing whether all versions got synced. This is being addressed in https://pulp.plan.io/issues/2594
"""

We may choose to pull in the fixes for https://pulp.plan.io/issues/2594 from 2.14 to 2.13 (or create a bz for that)

Comment 30 Brad Buckingham 2017-09-19 18:24:40 UTC
In order to verify this bug, really need to have access to the pulp fix in bug 1425605.  Without it, the behavior described in comment 27 will be observed.

Comment 32 pulp-infra@redhat.com 2017-11-28 15:04:31 UTC
Requesting needsinfo from upstream developer jortel because the 'FailedQA' flag is set.

Comment 34 pulp-infra@redhat.com 2017-11-28 22:41:47 UTC
Requesting needsinfo from upstream developer jortel because the 'FailedQA' flag is set.

Comment 37 pulp-infra@redhat.com 2017-11-29 16:04:54 UTC
Requesting needsinfo from upstream developer jortel because the 'FailedQA' flag is set.

Comment 39 pulp-infra@redhat.com 2017-12-13 20:04:37 UTC
Requesting needsinfo from upstream developer jortel because the 'FailedQA' flag is set.

Comment 41 Satellite Program 2017-12-13 21:15:33 UTC
Upstream bug assigned to paji

Comment 42 pulp-infra@redhat.com 2017-12-18 14:32:21 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 43 pulp-infra@redhat.com 2017-12-18 14:32:27 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 44 pulp-infra@redhat.com 2017-12-18 17:32:19 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 45 pulp-infra@redhat.com 2017-12-19 03:05:19 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 46 Satellite Program 2017-12-29 01:14:34 UTC
Upstream bug assigned to paji

Comment 47 Bryan Kearney 2018-01-03 17:12:11 UTC
*** Bug 1514009 has been marked as a duplicate of this bug. ***

Comment 48 pulp-infra@redhat.com 2018-01-09 09:02:11 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 49 pulp-infra@redhat.com 2018-01-09 09:31:40 UTC
Requesting needsinfo from upstream developer ttereshc because the 'FailedQA' flag is set.

Comment 50 Tanya Tereshchenko 2018-01-09 15:01:16 UTC
Importer was not updated properly in the certain cases. Fixed, see Pulp issue #3210.

An importer/distributor update is performed via asynchronous task, thus Katello should check if the task was finished successfully or not. As per comment 38, it seems like such check is not performed and Katello assumes that importer/distributor was updated successfully.

Comment 51 Satellite Program 2018-01-09 15:15:40 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/18076 has been resolved.

Comment 52 pulp-infra@redhat.com 2018-01-24 14:31:46 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 53 pulp-infra@redhat.com 2018-01-30 19:31:44 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 55 Partha Aji 2018-06-26 18:43:25 UTC
Hey Kedar,

Looks like the pulp change didn't make it to the 6.3 GA, but should be available 6.3.1 and above.

Comment 56 Kedar Bidarkar 2018-07-06 09:47:26 UTC
We can now perform OSTree Sync with "all History".

VERIFIED With Sat6.4.0-snap9

Comment 57 Kedar Bidarkar 2018-07-06 09:48:29 UTC
Created attachment 1456938 [details]
OSTree sync showing all branches of RHEL7

Comment 58 Kedar Bidarkar 2018-07-06 09:49:12 UTC
Created attachment 1456939 [details]
OSTree branches sync Coomplete

Comment 59 Bryan Kearney 2018-10-16 19:21:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2927


Note You need to log in before you can comment on or make changes to this bug.