Bug 1303546 - Atomic Storage Clients
Summary: Atomic Storage Clients
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: Changes Tracking
Version: 24
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Matt Micene
QA Contact:
URL:
Whiteboard: ChangeAcceptedF24, SelfContainedChange
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-02-01 09:33 UTC by Jan Kurik
Modified: 2016-09-29 11:25 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-09-29 11:25:53 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
Adds the required clients to the host manifest (697 bytes, patch)
2016-02-11 22:23 UTC, Matt Micene
no flags Details | Diff
Full rpm diff against 2016-01-05 21:40:09 23.45 98be0ae0f9 (1.71 KB, text/plain)
2016-02-12 15:48 UTC, Matt Micene
no flags Details
Full rpm diff against rawhide tree (1.29 KB, text/plain)
2016-02-12 17:16 UTC, Matt Micene
no flags Details
Full ceph diff against rawhide tree (1.08 KB, text/plain)
2016-02-12 18:47 UTC, Matt Micene
no flags Details
New patch removing ceph-fuse (640 bytes, patch)
2016-02-25 14:47 UTC, Matt Micene
no flags Details | Diff
Latest ceph diff against rawhide (472 bytes, text/plain)
2016-02-25 19:53 UTC, Matt Micene
no flags Details

Description Jan Kurik 2016-02-01 09:33:36 UTC
This is a tracking bug for Change: Atomic Storage Clients
For more details, see: https://fedoraproject.org//wiki/Changes/AtomicStorageClients

Kubernetes provides a mechanism for providing storage to Pods via volumes.  Volumes support several underlying storage protocols, but clients are needed to support each type.  Native GlusterFS and Ceph clients will be added to the Atomic host base to support these Volume types.

Comment 1 Matt Micene 2016-02-11 22:23:12 UTC
Created attachment 1123305 [details]
Adds the required clients to the host manifest

This patch updates the fedora-atomic-docker-host.json tree file to include the required packages for the GlusterFS and Ceph client support.

Ceph clients require ceph-common and ceph-fuse RPMs from the current release of Fedora.

Gluster clients require glusterfs and glusterfs-fuse from the current release of Fedora.

Testing of an updated Fedora 23 and Rawhide Atomic host was completed using the upstream Kubernetes examples found here:
https://github.com/kubernetes/kubernetes/tree/master/examples/glusterfs
https://github.com/kubernetes/kubernetes/tree/master/examples/rbd

Comment 2 Colin Walters 2016-02-12 15:06:58 UTC
Can you paste the RPM diff?  Did Ceph get fixed in Fedora to drop the redhat-lsb dep?

Comment 3 Matt Micene 2016-02-12 15:48:50 UTC
Created attachment 1123562 [details]
Full rpm diff against 2016-01-05 21:40:09     23.45     98be0ae0f9

Requested full output

Comment 4 Matt Micene 2016-02-12 16:00:54 UTC
Comment on attachment 1123562 [details]
Full rpm diff against 2016-01-05 21:40:09     23.45     98be0ae0f9

This list was created against F23 updates not Rawhide, will update with correct version

Comment 5 Matt Micene 2016-02-12 17:16:40 UTC
Created attachment 1123597 [details]
Full rpm diff against rawhide tree

This is the full rpm list from `rpm-ostree db diff` against a rawhide compose

Comment 6 Matt Micene 2016-02-12 17:54:36 UTC
(In reply to Colin Walters from comment #2)
> Can you paste the RPM diff?  Did Ceph get fixed in Fedora to drop the
> redhat-lsb dep?

Ceph in rawhide did drop the redhat-lsb dependency.  There are still a few dependencies pulled in for ceph like gdisk, hdparm, etc.  The attached list shows all packages added by the change.

I can break down by each set of clients if need be.

Comment 7 Matt Micene 2016-02-12 18:47:21 UTC
Created attachment 1123618 [details]
Full ceph diff against rawhide tree

This is the listing of ceph related additions to the tree

Comment 8 Colin Walters 2016-02-16 20:57:01 UTC
Why does ceph depend on python-flask?  Can we get some of the ceph developers onboard with this to try to trim this down a bit?

Comment 9 Daniel Walsh 2016-02-16 21:31:49 UTC
Now that we have docker (1.10) support for mount propagation.  We should be able to package all mount clients as containers going forward.

Comment 10 Matt Micene 2016-02-16 22:24:58 UTC
(In reply to Colin Walters from comment #8)
> Why does ceph depend on python-flask?  Can we get some of the ceph
> developers onboard with this to try to trim this down a bit?

It looks to me like this is part of the REST API implementation, I'll see if I can find a ceph developer to comment.

Comment 11 Jan Kurik 2016-02-24 14:26:12 UTC
On 2016-Feb-23, we have reached Fedora 24 Change Checkpoint: Completion deadline (testable).

At this point, all accepted changes should be substantially complete, and testable. Additionally, if a change is to be enabled by default, it must be so enabled at Change Completion deadline.

Change tracking bug should be set to the MODIFIED state to indicate it achieved completeness.

Incomplete and non testable Changes will be reported to FESCo on 2016-Feb-26 meeting.  Contingency plan for System Wide Changes, if planned for Alpha (or in case of serious doubts regarding Change completion), will be activated.

Comment 12 Ken Dreyer (Red Hat) 2016-02-24 18:53:02 UTC
(In reply to Matt Micene from comment #10)
> (In reply to Colin Walters from comment #8)
> > Why does ceph depend on python-flask?  Can we get some of the ceph
> > developers onboard with this to try to trim this down a bit?
> 
> It looks to me like this is part of the REST API implementation, I'll see if
> I can find a ceph developer to comment.

The Ceph server (ceph-mon) depends on python-flask; the client does not.

Comment 13 Matt Micene 2016-02-24 20:13:43 UTC
Confirmed with Ken that we only need ceph-common, not ceph-fuse.  ceph-fuse is pulling in ceph, which is the culprit for python-flask, etc.

Will recompose with just ceph-common and post a new list of rpms.

Comment 14 Ken Dreyer (Red Hat) 2016-02-24 20:45:36 UTC
We should fix the ceph-fuse requirement in any regard. http://tracker.ceph.com/issues/14854 . But yeah, ceph-common should be all you need.

Comment 15 Matt Micene 2016-02-25 14:47:10 UTC
Created attachment 1130577 [details]
New patch removing ceph-fuse

This is the latest patch which removes the addition of ceph-fuse

Comment 16 Daniel Walsh 2016-02-25 16:45:22 UTC
Are we packaging this up to run in a container or to run on atomic host directly?  And I am against installing it directly.

Comment 17 Colin Walters 2016-02-25 19:50:52 UTC
Dan and I had an in-person conversation about this.  There are multiple tensions here.  He feels specifically that by doing this we are disadvantaging 3rd party filesystems (or conversely special casing Ceph/Gluster etc.)

I wouldn't disagree, but OTOH most serious filesystems are going to use kernel drivers anyway, and we already special case that by having one kernel.

My feeling is that we need to execute in parallel on containerization and pressure relief via https://github.com/projectatomic/rpm-ostree/pull/107

Once we have that, things become a lot more flexible.

We also really need to come to some sort of fix for storage driver on non-Atomic Host, as using Docker containers on a non-AH system requires people to learn how to do new partitioning. In other words, it will be eaiser to drive containerization by default for Ceph and other things if we fix that.

Finally, one reason I'm leaning towards applying this is that it made it into the downstream EL release first, which should really never happen.  We want to be upstream first, and any fixes should land there first.  Since we don't have an out-of-the-box containerized Ceph now, we can't block on adding it to Fedora until we do.

Comment 18 Matt Micene 2016-02-25 19:53:04 UTC
Created attachment 1130649 [details]
Latest ceph diff against rawhide

Latest list of ceph-common only packages, down to 14 direct adds

Comment 19 Daniel Walsh 2016-02-25 20:40:12 UTC
Ok, I am fine with this, but I would really like people to experiment with the new features of docker-1.10 which allow mount propagation from within a container to affect the mount system on the host. We have spent a long time engineering this use case just so that remote file system mount clients can run inside of containers and be used on atomic host.

I would like to see Fedora leading the way in this department, and showing that we can run a minimal system for atomic host, and when we need additional software added, it comes in the form of a container image.

Comment 20 Matt Micene 2016-02-25 22:04:46 UTC
(In reply to Daniel Walsh from comment #19)
> Ok, I am fine with this, but I would really like people to experiment with
> the new features of docker-1.10 which allow mount propagation from within a
> container to affect the mount system on the host. We have spent a long time
> engineering this use case just so that remote file system mount clients can
> run inside of containers and be used on atomic host.
> 
> I would like to see Fedora leading the way in this department, and showing
> that we can run a minimal system for atomic host, and when we need
> additional software added, it comes in the form of a container image.

I'd like to see as much of a containerized infrastructure as we can, and like to see Fedora leading the way here too.  I also don't have a problem adding other 3rd party filesystems to alleviate any favoritism. That should hold for orchestration volume support.

I'd like to be as un-opinionated about the use of potential underlying storage systems for an Atomic Host as is reasonable.  The package layering approach makes this simpler (as far as I've wrapped my head around it) for Host / Docker storage locations, not a container / pod volume.

I think the new docker 1.10 features will make the orchestration layer drivers be more flexible. We're adding some but not all K8S volumes with this change.  But adding git just to support k8s gitRepo volumes I think is best suited for a container.

I don't think this change should be considered the be all end all of filesystem support but the best expedient at the moment.

Comment 21 Giuseppe Scrivano 2016-03-08 15:03:53 UTC
it seems possible to run Ceph in containers: https://hub.docker.com/u/ceph/

Should perhaps Ceph be used from containers instead of adding it to the host image?

Comment 22 Daniel Walsh 2016-03-08 15:09:23 UTC
I would prefer it to come from a container image.

Comment 23 Matt Micene 2016-03-10 13:56:54 UTC
I've been looking into a container solution, here's what I'm running into.  k8s volume support essentially boils down to exec'ing mount on the host.  If the host mount doesn't have the right support, mount fails, volume fails.

So mount support gets carried in ceph-common, which drops files in /usr/bin and /usr/lib.  This means (to me) that in order for a separate k8s container to use a separate ceph container, we need to present parts of the ceph container FS to the k8s container.  Individual volume mounts sounds painful, and I've not seen support for overlayfs between containers.

Unless someone knows of a good way to present a overlay, it seems like carrying the storage layer as part of a k8s container rather than a separate container is what we need to aim for.

Comment 24 Eric Paris 2016-04-06 00:14:55 UTC
Until kube 1.4 (at least) the only option is going to be 'on the host'. I know this stinks but there are just too many storage work items in kubernetes which come before containerized mount clients.

Docker 1.10 does appear to have laid the (adequate) ground work for kube to take advantage of. I believe non-kube users of 1.10 should have enough for containerized mount clients.

You can find the RH storage teams beginning thoughts and track the eventual effort to enable this at:

https://trello.com/c/gKJqcBRT/49-kubernetes-sidecar-storage-pattern-kube-volume-changes-gluster
(and other cards talking about "sidecar storage pattern"

I don't think there is any upstream community kubernetes effort at this time.

Comment 25 Colin Walters 2016-04-13 15:02:32 UTC
Ok, Matt, can you create a PR against https://pagure.io/fedora-atomic ?

Comment 26 Matt Micene 2016-04-20 14:55:34 UTC
(In reply to Colin Walters from comment #25)
> Ok, Matt, can you create a PR against https://pagure.io/fedora-atomic ?

PR created https://pagure.io/fedora-atomic/pull-request/2

Comment 27 Jan Kurik 2016-04-20 15:01:42 UTC
On 2016-Apr-19 we reached the "Change Checkpoint: 100% Code Complete Deadline" milestone for Fedora 24 release. At this point all the Changes not at least in in "ON_QA" state should be brought to FESCo for review. Please update the state of this bug to "ON_QA" if it is already 100% completed. Please let me know in case you have any trouble with the implementation and the Change needs any help or review.

Thanks, Jan

Comment 28 Colin Walters 2016-04-25 16:15:38 UTC
I took a look at this again.  One new issue is that ceph-common adds a `ceph` user, which we'll have to fixate.  It looks like ceph already reserved 167, so we can just add that to the `passwd` and `group`.

Comment 29 Matt Micene 2016-04-25 19:12:13 UTC
(In reply to Jan Kurik from comment #27)
> On 2016-Apr-19 we reached the "Change Checkpoint: 100% Code Complete
> Deadline" milestone for Fedora 24 release. At this point all the Changes not
> at least in in "ON_QA" state should be brought to FESCo for review. Please
> update the state of this bug to "ON_QA" if it is already 100% completed.
> Please let me know in case you have any trouble with the implementation and
> the Change needs any help or review.
> 
> Thanks, Jan

Sorry for the delay updating the status on the ticket. The PR has been committed in pagure and is completed.


Note You need to log in before you can comment on or make changes to this bug.