Created attachment 1685219 [details]
rook-ceph-operator logs

Description of problem:
When deploying OCS4.3 on a single-stack IPv4 OCP 4.4, the installation of the operator times out and some strange error messages appear in the logs of rook-ceph-operator (see attachment for full logs):

here is the alarming part:
2020-05-04 20:37:21.564322 I | exec: Running command: ceph orchestrator set backend rook --connect-timeout=15 --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --keyring=/var/lib/rook/openshift-storage/client.admin.keyring --format json --out-file /tmp/253648401
2020-05-04 20:37:21.849183 I | exec: no valid command found; 10 closest matches:
pg stat
pg getmap
pg dump {all|summary|sum|delta|pools|osds|pgs|pgs_brief [all|summary|sum|delta|pools|osds|pgs|pgs_brief...]}
pg dump_json {all|summary|sum|pools|osds|pgs [all|summary|sum|pools|osds|pgs...]}
pg dump_pools_json
pg ls-by-pool <poolstr> {<states> [<states>...]}
pg ls-by-primary <osdname (id|osd.id)> {<int>} {<states> [<states>...]}
pg ls-by-osd <osdname (id|osd.id)> {<int>} {<states> [<states>...]}
pg ls {<int>} {<states> [<states>...]}
pg dump_stuck {inactive|unclean|stale|undersized|degraded [inactive|unclean|stale|undersized|degraded...]} {<int>}
Error EINVAL: invalid command

Version-Release number of selected component (if applicable):
OCS 4.3.0 on OCP 4.4 bare-metal single-stack IPv6

How reproducible:
re-deployed this cluster multiple times, always the same result

Steps to Reproduce:
1. deploy OCP 4.4 single-stack IPv6 on bare-metal
2. deploy OCS according to the official doc (local-storage operator, then OCS)
3. installation of OCS never completes because the rook-ceph-operator never signal completion to the ocs-operator which in turns never signals he is healthy

Actual results:
installation stalls and timeout

Expected results:
installation completes with a functioning OCS cluster

Additional info:
I believe this is related to the fact that this cluster is running IPv6 single-stack, I cannot explain the invalid command in the logs but I don't see anything related to IPv6 in the ceph config file and I believe there should be something like "ms bind ipv6" in there.. see below for the config generated on a single-stack IPv6 cluster (note the absence of ms bind parameter):

[global]
fsid                     = 98a0cc4b-3318-4506-91c0-7b4fd145b00b
mon initial members      = a b c d e
mon host                 = v1:[2001:4958:a:3e00:0:1:3:18e6]:6789,v1:[2001:4958:a:3e00:0:1:3:5fab]:6789,v1:[2001:4958:a:3e00:0:1
:3:ac1d]:6789,v1:[2001:4958:a:3e00:0:1:3:ffae]:6789,v1:[2001:4958:a:3e00:0:1:3:f04c]:6789
mon keyvaluedb           = rocksdb
mon_allow_pool_delete    = true
mon_max_pg_per_osd       = 1000
debug default            = 0
debug rados              = 0
debug mon                = 0
debug osd                = 0
debug bluestore          = 0
debug filestore          = 0
debug journal            = 0
debug leveldb            = 0
filestore_omap_backend   = rocksdb
osd pg bits              = 11
osd pgp bits             = 11
osd pool default size    = 1
osd pool default pg num  = 100
osd pool default pgp num = 100
rbd_default_features     = 3
fatal signal handlers    = false
[client.admin]
keyring = /var/lib/rook/openshift-storage/client.admin.keyring