When using ceph-deploy 1.5.24 with pending ice-setup changes, the following repos get pre-loaded into /root/.cephdeploy.conf: ===== # Repositories [ceph-mon] name=Ceph-MON baseurl=http://admin.os1.phx2.redhat.com/static/MON gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release gpgcheck=0 default=true priority=1 proxy=_none_ [ceph-osd] name=Ceph-OSD baseurl=http://admin.os1.phx2.redhat.com/static/OSD gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release gpgcheck=0 default=true priority=1 proxy=_none_ ==== later, when doing "ceph-deploy install --repo <node>", there is no option to specify whether you want to install the repo file for ceph-mon or ceph-osd. It always takes the "default" repo, which is really just the first repo in the list that has "default=true" set. In this case, it is always ceph-mon. This means there is no way to push a .repo file to a node that defines ceph-osd. We need to be able to specify a repo name with "ceph-deploy install --repo" so that we can choose which repo to install. Additionally, it doesn't make any sense to have multiple repos with "default=true" set in cephdeploy.conf.
Good catch Travis. Yes, the 'default' key is meant to 'default to this one repo if I am not specifying anything' and it doesn't make sense to have multiple ones with 'default' in them. A side-effect of copying things around :( However, you can certainly have multiple repos installed at once or just one or the other. When specifying it overrides the 'default' key. If we want to install the 'ceph-osd' repository we need to specify it like: ceph-deploy install --repo --release=ceph-osd <node> I am inclined to say that this doesn't seem like a bug, but we would still need to update the docs/help to mention this. If we remove the default key from both the command will error. If we leave it for one, which one would it be? ceph-osd? or ceph-mon? Do we want to install both always? Installing both always as a default would require a template change for cephdeploy.conf only. Nothing would change for the user (I prefer this) but you would no longer be able to "just install ceph-osd" as it would pull the other one in.
Ah, I should have caught that the --release flag would do it. Sorry, it was getting late. :) Yeah, I'm really not sure which is the better answer. On the one hand, a lot more nodes will be OSD only than anything else, so in that sense it seems to me to make the most sense to make ceph-osd the default (if we keep them separate and have a default). Or, as you say, we could make neither the default and force the user to choose, or combine them into one .repo file (but with separate channels defined) and install both channels each time. I think we've trying to get away from having both mon and osd enabled automatically out of the box, so I would favor either making osd the default, or making the user choose. I wonder if this should be up to us, or who else we should ask.
After talking with Alfredo, we thought it made the most sense for the user to specify mon vs osd repos individually, and to remove the default. So, for a user setting up a node to run ceph-mon, they will need to: ceph-deploy install --repo --release=ceph-mon [nodes] ceph-deploy install --mon [nodes] And for setting up a node to run ceph-osd: ceph-deploy install --repo --release=ceph-osd [nodes] ceph-deploy install --osd [nodes] With the default repo removed, simply running "ceph-deploy install --repo [nodes]" will result in an error. This is all changes to how the cephdeploy config file is pre-configured by ice_setup, so changing subcomponent to RHC_setup.
This change has been tagged in ice-setup 0.4.0 upstream.
QE, in order to move this bug to VERIFIED, you should: 1) Ensure that your nodes do not have Puddle enabled (since this is testing ice-setup in an offline scenario). 2) Run ice_setup -d /mnt as per the docs. ...and then verify: 3) ice_setup prints the appropriate "ceph-deploy install" commands with the "--repo" and "--release" arguments to your console, 4) The "ceph-deploy install --repo" commands from #3 above succeed on your monitor nodes and OSD nodes, 5) The subsequent "ceph-deploy install --mon monitor1" and "ceph-deploy install --osd osd1" commands also succeed.
Verified with ISO Installation.. Moving to verified state
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2015:1183