Red Hat Bugzilla – Bug 1303546
Atomic Storage Clients
Last modified: 2016-09-29 07:25:53 EDT
This is a tracking bug for Change: Atomic Storage Clients
For more details, see: https://fedoraproject.org//wiki/Changes/AtomicStorageClients
Kubernetes provides a mechanism for providing storage to Pods via volumes. Volumes support several underlying storage protocols, but clients are needed to support each type. Native GlusterFS and Ceph clients will be added to the Atomic host base to support these Volume types.
Created attachment 1123305 [details]
Adds the required clients to the host manifest
This patch updates the fedora-atomic-docker-host.json tree file to include the required packages for the GlusterFS and Ceph client support.
Ceph clients require ceph-common and ceph-fuse RPMs from the current release of Fedora.
Gluster clients require glusterfs and glusterfs-fuse from the current release of Fedora.
Testing of an updated Fedora 23 and Rawhide Atomic host was completed using the upstream Kubernetes examples found here:
Can you paste the RPM diff? Did Ceph get fixed in Fedora to drop the redhat-lsb dep?
Created attachment 1123562 [details]
Full rpm diff against 2016-01-05 21:40:09 23.45 98be0ae0f9
Requested full output
Comment on attachment 1123562 [details]
Full rpm diff against 2016-01-05 21:40:09 23.45 98be0ae0f9
This list was created against F23 updates not Rawhide, will update with correct version
Created attachment 1123597 [details]
Full rpm diff against rawhide tree
This is the full rpm list from `rpm-ostree db diff` against a rawhide compose
(In reply to Colin Walters from comment #2)
> Can you paste the RPM diff? Did Ceph get fixed in Fedora to drop the
> redhat-lsb dep?
Ceph in rawhide did drop the redhat-lsb dependency. There are still a few dependencies pulled in for ceph like gdisk, hdparm, etc. The attached list shows all packages added by the change.
I can break down by each set of clients if need be.
Created attachment 1123618 [details]
Full ceph diff against rawhide tree
This is the listing of ceph related additions to the tree
Why does ceph depend on python-flask? Can we get some of the ceph developers onboard with this to try to trim this down a bit?
Now that we have docker (1.10) support for mount propagation. We should be able to package all mount clients as containers going forward.
(In reply to Colin Walters from comment #8)
> Why does ceph depend on python-flask? Can we get some of the ceph
> developers onboard with this to try to trim this down a bit?
It looks to me like this is part of the REST API implementation, I'll see if I can find a ceph developer to comment.
On 2016-Feb-23, we have reached Fedora 24 Change Checkpoint: Completion deadline (testable).
At this point, all accepted changes should be substantially complete, and testable. Additionally, if a change is to be enabled by default, it must be so enabled at Change Completion deadline.
Change tracking bug should be set to the MODIFIED state to indicate it achieved completeness.
Incomplete and non testable Changes will be reported to FESCo on 2016-Feb-26 meeting. Contingency plan for System Wide Changes, if planned for Alpha (or in case of serious doubts regarding Change completion), will be activated.
(In reply to Matt Micene from comment #10)
> (In reply to Colin Walters from comment #8)
> > Why does ceph depend on python-flask? Can we get some of the ceph
> > developers onboard with this to try to trim this down a bit?
> It looks to me like this is part of the REST API implementation, I'll see if
> I can find a ceph developer to comment.
The Ceph server (ceph-mon) depends on python-flask; the client does not.
Confirmed with Ken that we only need ceph-common, not ceph-fuse. ceph-fuse is pulling in ceph, which is the culprit for python-flask, etc.
Will recompose with just ceph-common and post a new list of rpms.
We should fix the ceph-fuse requirement in any regard. http://tracker.ceph.com/issues/14854 . But yeah, ceph-common should be all you need.
Created attachment 1130577 [details]
New patch removing ceph-fuse
This is the latest patch which removes the addition of ceph-fuse
Are we packaging this up to run in a container or to run on atomic host directly? And I am against installing it directly.
Dan and I had an in-person conversation about this. There are multiple tensions here. He feels specifically that by doing this we are disadvantaging 3rd party filesystems (or conversely special casing Ceph/Gluster etc.)
I wouldn't disagree, but OTOH most serious filesystems are going to use kernel drivers anyway, and we already special case that by having one kernel.
My feeling is that we need to execute in parallel on containerization and pressure relief via https://github.com/projectatomic/rpm-ostree/pull/107
Once we have that, things become a lot more flexible.
We also really need to come to some sort of fix for storage driver on non-Atomic Host, as using Docker containers on a non-AH system requires people to learn how to do new partitioning. In other words, it will be eaiser to drive containerization by default for Ceph and other things if we fix that.
Finally, one reason I'm leaning towards applying this is that it made it into the downstream EL release first, which should really never happen. We want to be upstream first, and any fixes should land there first. Since we don't have an out-of-the-box containerized Ceph now, we can't block on adding it to Fedora until we do.
Created attachment 1130649 [details]
Latest ceph diff against rawhide
Latest list of ceph-common only packages, down to 14 direct adds
Ok, I am fine with this, but I would really like people to experiment with the new features of docker-1.10 which allow mount propagation from within a container to affect the mount system on the host. We have spent a long time engineering this use case just so that remote file system mount clients can run inside of containers and be used on atomic host.
I would like to see Fedora leading the way in this department, and showing that we can run a minimal system for atomic host, and when we need additional software added, it comes in the form of a container image.
(In reply to Daniel Walsh from comment #19)
> Ok, I am fine with this, but I would really like people to experiment with
> the new features of docker-1.10 which allow mount propagation from within a
> container to affect the mount system on the host. We have spent a long time
> engineering this use case just so that remote file system mount clients can
> run inside of containers and be used on atomic host.
> I would like to see Fedora leading the way in this department, and showing
> that we can run a minimal system for atomic host, and when we need
> additional software added, it comes in the form of a container image.
I'd like to see as much of a containerized infrastructure as we can, and like to see Fedora leading the way here too. I also don't have a problem adding other 3rd party filesystems to alleviate any favoritism. That should hold for orchestration volume support.
I'd like to be as un-opinionated about the use of potential underlying storage systems for an Atomic Host as is reasonable. The package layering approach makes this simpler (as far as I've wrapped my head around it) for Host / Docker storage locations, not a container / pod volume.
I think the new docker 1.10 features will make the orchestration layer drivers be more flexible. We're adding some but not all K8S volumes with this change. But adding git just to support k8s gitRepo volumes I think is best suited for a container.
I don't think this change should be considered the be all end all of filesystem support but the best expedient at the moment.
it seems possible to run Ceph in containers: https://hub.docker.com/u/ceph/
Should perhaps Ceph be used from containers instead of adding it to the host image?
I would prefer it to come from a container image.
I've been looking into a container solution, here's what I'm running into. k8s volume support essentially boils down to exec'ing mount on the host. If the host mount doesn't have the right support, mount fails, volume fails.
So mount support gets carried in ceph-common, which drops files in /usr/bin and /usr/lib. This means (to me) that in order for a separate k8s container to use a separate ceph container, we need to present parts of the ceph container FS to the k8s container. Individual volume mounts sounds painful, and I've not seen support for overlayfs between containers.
Unless someone knows of a good way to present a overlay, it seems like carrying the storage layer as part of a k8s container rather than a separate container is what we need to aim for.
Until kube 1.4 (at least) the only option is going to be 'on the host'. I know this stinks but there are just too many storage work items in kubernetes which come before containerized mount clients.
Docker 1.10 does appear to have laid the (adequate) ground work for kube to take advantage of. I believe non-kube users of 1.10 should have enough for containerized mount clients.
You can find the RH storage teams beginning thoughts and track the eventual effort to enable this at:
(and other cards talking about "sidecar storage pattern"
I don't think there is any upstream community kubernetes effort at this time.
Ok, Matt, can you create a PR against https://pagure.io/fedora-atomic ?
(In reply to Colin Walters from comment #25)
> Ok, Matt, can you create a PR against https://pagure.io/fedora-atomic ?
PR created https://pagure.io/fedora-atomic/pull-request/2
On 2016-Apr-19 we reached the "Change Checkpoint: 100% Code Complete Deadline" milestone for Fedora 24 release. At this point all the Changes not at least in in "ON_QA" state should be brought to FESCo for review. Please update the state of this bug to "ON_QA" if it is already 100% completed. Please let me know in case you have any trouble with the implementation and the Change needs any help or review.
I took a look at this again. One new issue is that ceph-common adds a `ceph` user, which we'll have to fixate. It looks like ceph already reserved 167, so we can just add that to the `passwd` and `group`.
(In reply to Jan Kurik from comment #27)
> On 2016-Apr-19 we reached the "Change Checkpoint: 100% Code Complete
> Deadline" milestone for Fedora 24 release. At this point all the Changes not
> at least in in "ON_QA" state should be brought to FESCo for review. Please
> update the state of this bug to "ON_QA" if it is already 100% completed.
> Please let me know in case you have any trouble with the implementation and
> the Change needs any help or review.
> Thanks, Jan
Sorry for the delay updating the status on the ticket. The PR has been committed in pagure and is completed.