Description of problem: oc cluster up failed to start with logging install enabled. Version-Release number of selected component (if applicable): oc v3.10.0-alpha.0+73e9c8f-305 kubernetes v1.9.1+a0ce1bc657 features: Basic-Auth GSSAPI Kerberos SPNEGO How reproducible: always Steps to Reproduce: 1._output/local/bin/linux/amd64/oc cluster up --logging=true --loglevel=5 2. 3. Actual results: I0321 15:32:52.008624 17136 run.go:186] Container created with id "6e102f30d1f8454b5ffc81162f17d77e3d41cda892a1a10b574ea2f9e5e26213" I0321 15:32:52.008652 17136 run.go:255] Starting container "6e102f30d1f8454b5ffc81162f17d77e3d41cda892a1a10b574ea2f9e5e26213" I0321 15:32:52.660543 17136 run.go:262] Waiting for container "6e102f30d1f8454b5ffc81162f17d77e3d41cda892a1a10b574ea2f9e5e26213" I0321 15:32:53.736175 17136 run.go:268] Done waiting for container "6e102f30d1f8454b5ffc81162f17d77e3d41cda892a1a10b574ea2f9e5e26213", rc=0 I0321 15:32:53.736201 17136 run.go:273] Reading logs from container "6e102f30d1f8454b5ffc81162f17d77e3d41cda892a1a10b574ea2f9e5e26213" I0321 15:32:53.760081 17136 run.go:281] Done reading logs from container "6e102f30d1f8454b5ffc81162f17d77e3d41cda892a1a10b574ea2f9e5e26213" I0321 15:32:53.760105 17136 run.go:283] Stdout: openshift v3.10.0-alpha.0+73e9c8f-305 kubernetes v1.9.1+a0ce1bc657 etcd 3.2.16 I0321 15:32:53.760119 17136 run.go:284] Stderr: I0321 15:32:53.760132 17136 run.go:288] Container run successful I0321 15:32:53.760141 17136 run.go:244] Deleting container "6e102f30d1f8454b5ffc81162f17d77e3d41cda892a1a10b574ea2f9e5e26213" I0321 15:32:55.663315 17136 ansible.go:191] Generated ansible inventory: [OSEv3:children] masters nodes [OSEv3:vars] #openshift_release=v3.10 openshift_deployment_type=origin openshift_logging_image_prefix=openshift/origin- openshift_logging_image_version=v3.10 openshift_logging_master_public_url=https://127.0.0.1:8443 openshift_logging_install_logging=true openshift_logging_use_ops=false openshift_logging_namespace=logging openshift_logging_elasticseach_memory_limit=1024M openshift_logging_elasticseach_storage_type=pvc openshift_logging_elasticseach_pvc_size=100G openshift_logging_kibana_hostname=kibana-logging.127.0.0.1.nip.io [masters] 127.0.0.1 ansible_connection=local [nodes] 127.0.0.1 [etcd] 127.0.0.1 I0321 15:32:55.663969 17136 ansible.go:161] Wrote inventory to local file: /tmp/openshift-inventory431480227 I0321 15:32:55.663993 17136 ansible.go:163] Uploading file /tmp/openshift-inventory431480227 to host destination: /var/lib/origin/openshift.local.config/logging.inventory I0321 15:32:55.664020 17136 run.go:181] Creating container named "" config: image: openshift/origin:v3.10 entry point: /bin/true host config: pid mode: user mode: network mode: volume binds: :/var/lib/origin/openshift.local.config:z Error: FAIL Error: cannot create container using image openshift/origin:v3.10 Caused By: Error: Error response from daemon: invalid bind mount spec ":/var/lib/origin/openshift.local.config:z": invalid volume specification: ':/var/lib/origin/openshift.local.config:z' Expected results: succeed Additional info:
Jeff: it looks like the experimental logging and metrics flags were trying to modify the running container that used to run the all-in-one. The `origin` container is now just a kubelet and nothing should be modifying it. Every component installing into cluster up uses its own container or runs directly on the host. They do not exec into the kubelet or attempt to modify the kubelet. You may need to refactor some approaches to handle this, but the flags are currently experimental so you'll have some leeway. Our short to mid-term vision for cluster up has it finishing it's work at the point when the openshift APIs are available. Keep that in mind as a need to avoid glue code that affects multiple components.
@Eric: We can 'docker run' the openshift-ansible container with the proper inventory and playbook like what is done in the working version of 'oc cluster up'. The one concern I have is that currently, we try to modify the config to add in the kibana url (or hawkular for metrics) and restart the api server. I assume we will need to either disable (make it conditional) that functionality for pre-3.10 or use the new configmap approach for 3.10+
Moving to 3.11 as our friends on the broker team are going to fix it up to use the new pattern.
I've submitted a PR which should get logging working again. https://github.com/openshift/origin/pull/19823
Moving to New and 4.1 as per @deads 'oc cluster up' is broken in 4.0. May be resolved as a by-product of cluster-logging-operator being added to stack.
Closing as 'oc cluster up' was removed in 4.0