Dave would you please provide the output from `rpm -qv ceph-ansible ansible` ?

(In reply to Ken Dreyer (Red Hat) from comment #2)
> Dave would you please provide the output from `rpm -qv ceph-ansible ansible`
> ?

(undercloud) [stack@microbrow-07 ~]$ rpm -qv ceph-ansible ansible
ceph-ansible-3.0.0-0.1.rc4.el7cp.noarch
ansible-2.3.2.0-2.el7.noarch

The problem is the name of the prepare container comes from the 'devices' list.
We need to resolve the link first I suppose.

It's worse than that.  If you resolve the link to a device name, that will be wrong when system reboots.  For example, this problem impacts the scale lab SuperMicro 6048R hosts that have 2 storage controllers, an onboard controller that manages the internal system-disk drives, and an LSI 3108 that manages the 2 storage shelves with 36 drives.  Even when the LSI 3108 is not in JBOD mode and has virtual drive defined for each HDD, it will not come up with the same device name assignment every time because of the multiple controllers.

Could we resolve the softlink to a WWID and use that instead to name the container?  At least that name won't change when the system reboots, and it can be quickly resolved to the device name du jour using /dev/disk/by-id/wwn- softlink.  Also, this fits the model of hot-pluggable Ceph OSDs.

Also, how would this work if LVM volume was used?  In that case it would be best to specify the container by some transformation of its LVM logical volume pathname, which includes the group name, since this again is stable across reboots and should be hot-pluggable as well.

Ben,

Resolving the links is not a problem, even if they change across reboots. If sda becomes sdb after reboot, it's still the same osd inside so it will run.

Actually, the only problem with the approach is a potential collision with the root disk of the server. However this is not really an issue, the activation will simply fail.

will be in rc15

(In reply to leseb from comment #8)
> Actually, the only problem with the approach is a potential collision with
> the root disk of the server. However this is not really an issue, the
> activation will simply fail.

What we see in some of the supermicros with 2 controllers is it's inconsistent on the device mapping between controllers. So, say the system disk is originally mapped as sdaa along with 3 other non-osd disks:sdab,sdac,sdad( with osp we assign root disk to wwid so there will be no conflict) those other disks on reboot
 would be sdb,sdc,sdd which would be instantiated as container osds but are not.

This is still broken with docker.io/ceph/daemon:tag-build-race-jewel-devel-wip-bz1494543-jewel-kefu-centos-7 and ceph-ansible-3.0.0-0.rc18.1.el7.noarch.

Oct 10 12:10:29 localhost dockerd-current: time="2017-10-10T12:10:29.852074497-04:00" level=error msg="Handler for POST /v1.24/containers/create returned error: Invalid container name (ceph-osd-prepare-overcloud-cephstorage-0-disk/by-path/pci-0000:03:00.0-scsi-0:0:10:0), only [a-zA-Z0-9][a-zA-Z0-9_.-] are allowed"

How to reproduce this issue:

Do a containerized deployment but when you specify your device list, instead of using: 

devices:
- /dev/vdb
- /dev/vdc

Instead use: 

devices:
- /dev/disk/by-path/virtio-pci-0000:00:06.0
- /dev/disk/by-path/virtio-pci-0000:00:07.0

The above are alternative paths to the same device. How do you find the alternative paths on your system? Any RHEL box will have them, just look in the /dev/disk/by-path/ directory. For example I can see they're the same on my box as per: 

[root@overcloud-cephstorage-0 ~]# ls -l /dev/disk/by-path/virtio-pci-0000\:00\:06.0*
lrwxrwxrwx. 1 root root  9 Oct 18 14:43 /dev/disk/by-path/virtio-pci-0000:00:06.0 -> ../../vdb
lrwxrwxrwx. 1 root root 10 Oct 18 14:43 /dev/disk/by-path/virtio-pci-0000:00:06.0-part1 -> ../../vdb1
[root@overcloud-cephstorage-0 ~]# ls -l /dev/disk/by-path/virtio-pci-0000\:00\:07.0*
lrwxrwxrwx. 1 root root  9 Oct 18 14:43 /dev/disk/by-path/virtio-pci-0000:00:07.0 -> ../../vdc
lrwxrwxrwx. 1 root root 10 Oct 18 14:43 /dev/disk/by-path/virtio-pci-0000:00:07.0-part1 -> ../../vdc1
[root@overcloud-cephstorage-0 ~]#

This is definitely working on 3.0, I'm even surprise rc18 failed, I'm sure the fix is in it.

(In reply to leseb from comment #21)
> This is definitely working on 3.0, I'm even surprise rc18 failed, I'm sure
> the fix is in it.

Dave had run into it but I see it's in your CI so it must have passed. I will test it myself with ceph-ansible-3.0.2-1.el7cp.noarch as an extra test and update this bug.

Indeed, we test this in our CI now. Thanks John.

Dave tested it again and hit the issue. On his next deploy I'll see if I can get on the node during the failure and run some ceph-docker commands to get more info. 

For now here is what I see so far:

The logs from the playbook run on Dave's machine indicate that he got the Invalid container name error [0]. I see the patch in question [1][2] applied from ceph-ansible-3.0.3-1.el7cp.noarch. I see that the playbook was called with the following for the devices list as per [3]. 

"devices": ["/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:10:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:11:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:12:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:13:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:14:0"]


Footnotes:

[0] 

[root@microbrow-07 mistral]# grep -A 1 "ceph-osd : prepare ceph \"filestore\" containerized osd disk(s) non-collocated" ceph-install-workflow.log 
...
2017-10-19 16:07:11,719 p=28199 u=mistral |  TASK [ceph-osd : prepare ceph "filestore" containerized osd disk(s) non-collocated] ***
2017-10-19 16:07:13,354 p=28199 u=mistral |  failed: [192.168.24.62] (item=[{'_ansible_parsed': True, 'stderr_lines': [], '_ansible_item_result': True, u'end': u'2017-10-19 20:06:56.049731', '_ansible_no_log': False, u'stdout': u'', u'cmd': u"parted --script /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:10:0 print | egrep -sq '^ 1.*ceph'", u'rc': 1, 'item': u'/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:10:0', u'delta': u'0:00:00.007669', u'stderr': u'', u'changed': False, u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': True, u'_raw_params': u"parted --script /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:10:0 print | egrep -sq '^ 1.*ceph'", u'removes': None, u'creates': None, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2017-10-19 20:06:56.042062', 'failed': False}, u'/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:10:0', u'/dev/nvme0n1']) => {"changed": true, "cmd": "docker run --net=host --rm --pid=host --privileged=true --name=ceph-osd-prepare-overcloud-cephstorage-0-disk/by-path/pci-0000:03:00.0-scsi-0:0:10:0 -v /etc/ceph:/etc/ceph -v /var/lib/ceph/:/var/lib/ceph/ -v /dev:/dev -v /etc/localtime:/etc/localtime:ro -e CLUSTER=ceph -e CEPH_DAEMON=OSD_CEPH_DISK_PREPARE -e OSD_DEVICE=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:10:0 -e OSD_JOURNAL=/dev/nvme0n1 -e OSD_BLUESTORE=0 -e OSD_FILESTORE=1 -e OSD_DMCRYPT=0 -e OSD_JOURNAL_SIZE=5120 docker-registry.engineering.redhat.com/ceph/rhceph-2-rhel7:latest", "delta": "0:00:00.025579", "end": "2017-10-19 20:07:13.325883", "failed": true, "item": [{"_ansible_item_result": true, "_ansible_no_log": false, "_ansible_parsed": true, "changed": false, "cmd": "parted --script /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:10:0 print | egrep -sq '^ 1.*ceph'", "delta": "0:00:00.007669", "end": "2017-10-19 20:06:56.049731", "failed": false, "failed_when_result": false, "invocation": {"module_args": {"_raw_params": "parted --script /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:10:0 print | egrep -sq '^ 1.*ceph'", "_uses_shell": true, "chdir": null, "creates": null, "executable": null, "removes": null, "warn": true}}, "item": "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:10:0", "rc": 1, "start": "2017-10-19 20:06:56.042062", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}, "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:10:0", "/dev/nvme0n1"], "rc": 125, "start": "2017-10-19 20:07:13.300304", "stderr": "/usr/bin/docker-current: Error response from daemon: Invalid container name (ceph-osd-prepare-overcloud-cephstorage-0-disk/by-path/pci-0000:03:00.0-scsi-0:0:10:0), only [a-zA-Z0-9][a-zA-Z0-9_.-] are allowed.\nSee '/usr/bin/docker-current run --help'.", "stderr_lines": ["/usr/bin/docker-current: Error response from daemon: Invalid container name (ceph-osd-prepare-overcloud-cephstorage-0-disk/by-path/pci-0000:03:00.0-scsi-0:0:10:0), only [a-zA-Z0-9][a-zA-Z0-9_.-] are allowed.", "See '/usr/bin/docker-current run --help'."], "stdout": "", "stdout_lines": []}


[1] 
https://github.com/ceph/ceph-ansible/pull/1956/commits/3bd341f6c0faddbb62a1950869069d2491c5be55#diff-bc1034a6fab2180f375254c274e02ed6

[2] 
[root@microbrow-07 mistral]# grep -B 10 regex_replace /usr/share/ceph-ansible/roles/ceph-osd/tasks/scenarios/non-collocated.yml
---
# use shell rather than docker module
# to ensure osd disk prepare finishes before
# starting the next task
- name: prepare ceph "{{ osd_objectstore }}" containerized osd disk(s) non-collocated
  shell: |
    docker run --net=host \
    --rm \
    --pid=host \
    --privileged=true \
    --name=ceph-osd-prepare-{{ ansible_hostname }}-{{ item.1 | regex_replace('/dev/', '') }} \
--
    - osd_objectstore == 'filestore'
    - not item.0.get("skipped")
    - item.0.get("rc", 0) != 0

- name: prepare ceph "{{ osd_objectstore }}" containerized osd disk(s) non-collocated with a dedicated device for db and wal
  shell: |
    docker run --net=host \
    --rm \
    --pid=host \
    --privileged=true \
    --name=ceph-osd-prepare-{{ ansible_hostname }}-{{ item.1 | regex_replace('/dev/', '') }} \
[root@microbrow-07 mistral]# 

[3]

[root@microbrow-07 mistral]#  grep site-docker.yml.sample executor.log | grep ansible-playbook | tail -1 
2017-10-19 16:07:19.804 16420 ERROR mistral.executors.default_executor Command: ansible-playbook /usr/share/ceph-ansible/site-docker.yml.sample --user tripleo-admin --become --become-user root --extra-vars {"monitor_secret": "***", "ceph_conf_overrides": {"global": {"rgw_s3_auth_use_keystone": "true", "rgw_keystone_admin_password": "***", "rgw_keystone_url": "http://172.16.0.16:5000", "rgw_keystone_admin_project": "service", "rgw_keystone_accepted_roles": "Member, _member_, admin", "osd_pool_default_size": 1, "osd_pool_default_pg_num": 1024, "rgw_keystone_api_version": 3, "rgw_keystone_admin_user": "swift", "rgw_keystone_admin_domain": "default"}}, "osd_scenario": "non-collocated", "fetch_directory": "/tmp/file-mistral-actionrALzJT", "user_config": true, "ceph_docker_image_tag": "latest", "ceph_release": "jewel", "containerized_deployment": true, "fsid": "e8b51a76-a24a-11e7-9b56-0cc47a527198", "public_network": "172.18.0.0/24", "generate_fsid": false, "monitor_address_block": "172.18.0.0/24", "monitor_interface": "vlan302", "admin_secret": "***", "keys": [{"mon_cap": "allow r", "osd_cap": "allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics", "mode": "0644", "key": "AQDrk8lZAAAAABAAMti2BVr55DLFPZtF81orwQ==", "name": "client.openstack"}, {"mon_cap": "allow r, allow command \\\\\\\"auth del\\\\\\\", allow command \\\\\\\"auth caps\\\\\\\", allow command \\\\\\\"auth get\\\\\\\", allow command \\\\\\\"auth get-or-create\\\\\\\"", "mds_cap": "allow *", "name": "client.manila", "mode": "0644", "key": "AQDrk8lZAAAAABAA9iUoZeIZguIXIgvErdxEIA==", "osd_cap": "allow rw"}, {"mon_cap": "allow rw", "osd_cap": "allow rwx", "mode": "0644", "key": "AQDrk8lZAAAAABAA0cHSyGZr16ZehPHA/QDg6A==", "name": "client.radosgw"}], "openstack_keys": [{"mon_cap": "allow r", "osd_cap": "allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics", "mode": "0644", "key": "AQDrk8lZAAAAABAAMti2BVr55DLFPZtF81orwQ==", "name": "client.openstack"}, {"mon_cap": "allow r, allow command \\\\\\\"auth del\\\\\\\", allow command \\\\\\\"auth caps\\\\\\\", allow command \\\\\\\"auth get\\\\\\\", allow command \\\\\\\"auth get-or-create\\\\\\\"", "mds_cap": "allow *", "name": "client.manila", "mode": "0644", "key": "AQDrk8lZAAAAABAA9iUoZeIZguIXIgvErdxEIA==", "osd_cap": "allow rw"}, {"mon_cap": "allow rw", "osd_cap": "allow rwx", "mode": "0644", "key": "AQDrk8lZAAAAABAA0cHSyGZr16ZehPHA/QDg6A==", "name": "client.radosgw"}], "osd_objectstore": "filestore", "pools": [], "ntp_service_enabled": false, "ceph_docker_image": "ceph/rhceph-2-rhel7", "cluster_network": "172.19.0.0/24", "dedicated_devices": ["/dev/nvme0n1", "/dev/nvme0n1", "/dev/nvme0n1", "/dev/nvme0n1", "/dev/nvme0n1"], "journal_size": 5120, "openstack_config": true, "ceph_docker_registry": "docker-registry.engineering.redhat.com", "ceph_stable": true, "devices": ["/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:10:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:11:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:12:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:13:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:14:0"], "ceph_origin": "distro", "openstack_pools": [{"rule_name": "", "pg_num": 1024, "name": "volumes"}, {"rule_name": "", "pg_num": 1024, "name": "backups"}, {"rule_name": "", "pg_num": 1024, "name": "vms"}, {"rule_name": "", "pg_num": 1024, "name": "images"}, {"rule_name": "", "pg_num": 1024, "name": "metrics"}], "ip_version": "ipv4", "ireallymeanit": "yes", "docker": true} --forks 4 --ssh-common-args "-o StrictHostKeyChecking=no" --ssh-extra-args "-o UserKnownHostsFile=/dev/null" --inventory-file /tmp/ansible-mistral-actionSs2x6l/inventory.yaml --private-key /tmp/ansible-mistral-actionSs2x6l/ssh_private_key --skip-tags package-install,with_pkg
[root@microbrow-07 mistral]#

I've reproduced this in my environment too. I opened the following upstream issue and will look into a follow up patch: 

 https://github.com/ceph/ceph-ansible/issues/2094

John, the current state is unclear to me; do we need more code changes for this BZ?

(In reply to Ken Dreyer (Red Hat) from comment #26)
> John, the current state is unclear to me; do we need more code changes for
> this BZ?

Ken, based on my testing yes, I think we do need more code changes. My plan is to submit a PR with the new github issues to prove the difference (once I get it working with the combination reported in the upstream issue). I've done a needinfo to Seb so he can comment before the status is changed. 

  John

FYI, I'm still unable to reproduce this issue, Guillaume is looking into this and will provide more info soon.