Attachment 1957836 Details for Bug 2187305 – attached the testcase of the failure

attached the testcase of the failure

test_pvc_evict_ceph_clients-same.log (text/plain), 133.72 KB, created by Abhishek Mundada on 2023-04-17 12:18:34 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Abhishek Mundada

Created: 2023-04-17 12:18:34 UTC

Size: 133.72 KB

patch

obsolete

>============================= test session starts ==============================
>platform linux -- Python 3.9.14, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
>rootdir: /root/ocs-upi-kvm/src/ocs-ci, configfile: pytest.ini
>plugins: flaky-3.7.0, repeat-0.9.1, ordering-0.6, metadata-1.11.0, logger-0.5.1, marker-bugzilla-0.9.4, html-3.1.1
>collected 1 item
>
>tests/manage/pv_services/test_pvc_evict_ceph_clients.py::TestPvcEvictCephClients::test_pvc_evict_ceph_clients[same] 
>-------------------------------- live log setup --------------------------------
>02:08:01 - MainThread - ocs_ci.utility.utils - INFO  - testrun_name: OCS4-13-Downstream-OCP4-13-POWERVS-UPI-1AZ-RHCOS-LSO-3M-3W-tier2
>02:08:01 - MainThread - ocs_ci.utility.utils - INFO  - testrun_name: OCS4-13-Downstream-OCP4-13-POWERVS-UPI-1AZ-RHCOS-LSO-3M-3W-tier2
>02:08:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc version -o json
>02:08:01 - MainThread - ocs_ci.utility.utils - INFO  - Retrieving the authentication config dictionary
>02:08:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get pods -o name
>02:08:01 - MainThread - ocs_ci.ocs.utils - INFO  - pod name match found appending rook-ceph-tools-c8d94fcbd-cx8xl
>02:08:01 - MainThread - ocs_ci.ocs.utils - INFO  - Ceph toolbox already exists, skipping
>02:08:01 - MainThread - tests.conftest - INFO  - All logs located at /root/logs-ocs-ci/4.13/ocs-ci-logs-1680674868
>02:08:01 - MainThread - tests.conftest - INFO  - Skipping client download
>02:08:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get csv  -n openshift-storage -o yaml
>02:08:03 - MainThread - ocs_ci.ocs.version - INFO  - collecting ocp version
>02:08:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get clusterversion version -o yaml
>02:08:03 - MainThread - ocs_ci.ocs.version - INFO  - collecting ocs version
>02:08:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get namespace  -o yaml
>02:08:03 - MainThread - ocs_ci.ocs.version - INFO  - found storage namespaces ['openshift-cluster-storage-operator', 'openshift-kube-storage-version-migrator', 'openshift-kube-storage-version-migrator-operator', 'openshift-local-storage', 'openshift-storage']
>02:08:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-cluster-storage-operator get pod  -n openshift-cluster-storage-operator -o yaml
>02:08:04 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-kube-storage-version-migrator get pod  -n openshift-kube-storage-version-migrator -o yaml
>02:08:04 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-kube-storage-version-migrator-operator get pod  -n openshift-kube-storage-version-migrator-operator -o yaml
>02:08:04 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-local-storage get pod  -n openshift-local-storage -o yaml
>02:08:04 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get pod  -n openshift-storage -o yaml
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - ClusterVersion .spec.channel: stable-4.12
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - ClusterVersion .status.desired.version: 4.13.0-0.nightly-ppc64le-2023-03-25-100749
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - ClusterVersion .status.desired.image: sys-powercloud-docker-local.artifactory.swg-devops.com/ocp-ppc64le/release-ppc64le:4.13.0-0.nightly-ppc64le-2023-03-25-100749
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - storage namespace openshift-cluster-storage-operator
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b1775489b52368d38a312cbb34e4b260c2a50666fb3c3a71787bf9a9e9f9ecf {'quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b1775489b52368d38a312cbb34e4b260c2a50666fb3c3a71787bf9a9e9f9ecf'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15d5742b3e2b0feb61d088b6811f3303b8d4f3dcfe060087ad74930623cf505c {'quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15d5742b3e2b0feb61d088b6811f3303b8d4f3dcfe060087ad74930623cf505c'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e2a88c25095508fed6a75f17898e0ba1e175553ec0fdf83c2aeabb596f8bea6 {'quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e2a88c25095508fed6a75f17898e0ba1e175553ec0fdf83c2aeabb596f8bea6'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fbe78daa5c2a93f45f9a18135696060ba38a5ef1c3bd2a94f720be0d7702cea7 {'quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fbe78daa5c2a93f45f9a18135696060ba38a5ef1c3bd2a94f720be0d7702cea7'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - storage namespace openshift-kube-storage-version-migrator
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b654feeace8e3ec0f582125884bf6d1a0bbe7a39111d9d46bef2281b7adf46bc {'quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b654feeace8e3ec0f582125884bf6d1a0bbe7a39111d9d46bef2281b7adf46bc'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - storage namespace openshift-kube-storage-version-migrator-operator
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82afda6a7af6ef631dd975ea0b6b155ab2ddca38553b2225393a36ae4137fb65 {'quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82afda6a7af6ef631dd975ea0b6b155ab2ddca38553b2225393a36ae4137fb65'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - storage namespace openshift-local-storage
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/openshift4/ose-local-storage-diskmaker@sha256:96eb5ace968ba319ad96a0675a9a282f739c2cf4cfcec3881a2e19a97759b93c {'registry.redhat.io/openshift4/ose-local-storage-diskmaker@sha256:49c9e0b388d4edd20f91ea252748648b858b7ba7a1924a4ecbdf10d6f97b0644'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:d235ec8c9e87b21cdc4c42ba165a86ac3abc346533efc27b97dd7d48b5e80bb8 {'registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:06ba20876c01aef803be9e3aaf4c0d270d9e292cf6b4896ec95c1f3a74923696'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image registry.redhat.io/openshift4/ose-local-storage-operator@sha256:c4f58c6f98ea63ef0c0d6f5a0d305286ad9cae19c0225c77703d41eeaa959374 {'registry.redhat.io/openshift4/ose-local-storage-operator@sha256:183f29b2d6cd0cae149fd00ca8d79ba775e0c2cf27facdc1cc0a0e5e3833de9a'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - storage namespace openshift-storage
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/rhceph-dev/openshift-ose-kube-rbac-proxy@sha256:d291e0d38bd95eb2731a10bd5ff06bfa07c6621528d3c2bbe888397d87b3feaa {'quay.io/rhceph-dev/openshift-ose-kube-rbac-proxy@sha256:d291e0d38bd95eb2731a10bd5ff06bfa07c6621528d3c2bbe888397d87b3feaa'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/rhceph-dev/odf4-odf-csi-addons-rhel9-operator@sha256:707735dff22e42706c5b09fd63a19f0917e836859c0e790dac507959ceef6ccc {'quay.io/rhceph-dev/odf4-odf-csi-addons-rhel9-operator@sha256:000bcce1fce7fd25a312e6d48d73fccf2a393d164743794246de20eb7772aa23'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/rhceph-dev/odf4-cephcsi-rhel9@sha256:3c8f2dfa28ba2a60e6bcda9f0d57a96154376d9286db572b426756ea39defa40 {'quay.io/rhceph-dev/odf4-cephcsi-rhel9@sha256:3c8f2dfa28ba2a60e6bcda9f0d57a96154376d9286db572b426756ea39defa40'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/rhceph-dev/openshift-ose-csi-node-driver-registrar@sha256:ad95c08249b6ee11792b1b7e1d223551b38240a1c6194b308f81be591ce62d98 {'quay.io/rhceph-dev/openshift-ose-csi-node-driver-registrar@sha256:22326d3d5438b94adf33de2d3d484c12ceb7c9d358adc5c3552b0740bcc824ef'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/rhceph-dev/openshift-ose-csi-external-attacher@sha256:0c557ab7afdeabd0b4af3ddc0c969ca06f1c6dd758b8eb1c56a38133f4510068 {'quay.io/rhceph-dev/openshift-ose-csi-external-attacher@sha256:0c557ab7afdeabd0b4af3ddc0c969ca06f1c6dd758b8eb1c56a38133f4510068'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/rhceph-dev/openshift-ose-csi-external-provisioner@sha256:74d81a9f06ccd9309226e845f0dfdd21fd2341155e7307b607d03053aff01eba {'quay.io/rhceph-dev/openshift-ose-csi-external-provisioner@sha256:52fb45272a5f140a9e2f7c3e5a08b9666ae7408dcb7efbe48100f1d9b3541984'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/rhceph-dev/openshift-ose-csi-external-resizer@sha256:12961c8858c7e68cefe54de0379c0f1a2663c33f94343555a606a9ab086cc50a {'quay.io/rhceph-dev/openshift-ose-csi-external-resizer@sha256:12961c8858c7e68cefe54de0379c0f1a2663c33f94343555a606a9ab086cc50a'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/rhceph-dev/openshift-ose-csi-external-snapshotter@sha256:de91050fa4f04ab33f8b3565daedaf2a1af3921a434e69c7a3a6942331109147 {'quay.io/rhceph-dev/openshift-ose-csi-external-snapshotter@sha256:1c54ae31f5a09449da0d2bdeb7bfafa03c078e70de7ea2e740ef0ed13cf37540'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/rhceph-dev/odf4-odf-csi-addons-sidecar-rhel9@sha256:2689ee3c9a945d3325d605a15ba4e39dad8cacbbb3cbb2afe518bfa73f637160 {'quay.io/rhceph-dev/odf4-odf-csi-addons-sidecar-rhel9@sha256:2689ee3c9a945d3325d605a15ba4e39dad8cacbbb3cbb2afe518bfa73f637160'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/rhceph-dev/odf4-mcg-core-rhel9@sha256:08e5ae2e9869d434bd7d522dd81bb9900c71b05d63bbfde474d8d2250dd5fae9 {'quay.io/rhceph-dev/odf4-mcg-core-rhel9@sha256:08e5ae2e9869d434bd7d522dd81bb9900c71b05d63bbfde474d8d2250dd5fae9'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/rhceph-dev/rhel8-postgresql-12@sha256:e88838e08efabb25d2450f3c178ffe0a9e1be1a14c5bc998bde0e807c9a3f15f {'quay.io/rhceph-dev/rhel8-postgresql-12@sha256:6ffbd09fc53f102a078f2000762680c4b5bb60d283f4d9b4977f00b4fd8abb66'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/rhceph-dev/odf4-mcg-rhel9-operator@sha256:56116a77ec6933e729dd9920bddf1ebd6a88605e892783d263900f254f25a3d1 {'quay.io/rhceph-dev/odf4-mcg-rhel9-operator@sha256:09ded8aec454b5d0660c196cf9f9ac3e8c0c60bae687781794e2c7e4641bd599'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/rhceph-dev/odf4-ocs-metrics-exporter-rhel9@sha256:538dd263db25cafd82a4e967731ce26c8f3a8e23732bba54baf399ea3ab56394 {'quay.io/rhceph-dev/odf4-ocs-metrics-exporter-rhel9@sha256:538dd263db25cafd82a4e967731ce26c8f3a8e23732bba54baf399ea3ab56394'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/rhceph-dev/odf4-ocs-rhel9-operator@sha256:c0557b0a2db726d72b80aa0fe0a46e49c9a4f67a2316b05e0621dcd73e25b0fb {'quay.io/rhceph-dev/odf4-ocs-rhel9-operator@sha256:33acd985d3c3024ac5c509473aa4435d446827597fb723e70d7068bb531bd1fd'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/rhceph-dev/odf4-odf-console-rhel9@sha256:217ecb352ddb265e9b64b0d31ffa71467729244a846941021847bd936fa8f1c5 {'quay.io/rhceph-dev/odf4-odf-console-rhel9@sha256:217ecb352ddb265e9b64b0d31ffa71467729244a846941021847bd936fa8f1c5'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/rhceph-dev/odf4-odf-rhel9-operator@sha256:fe2911cff0f55ac61274b9404ba527f72a27b125cde854ebeb41551a0aafdeae {'quay.io/rhceph-dev/odf4-odf-rhel9-operator@sha256:aa303840c9d1c7649a4fec43e5f5a95c9f43e984d077328cf0a62f55e14beea6'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/rhceph-dev/rhceph@sha256:f916da02f59b8f73ad18eb65310333d1e3cbd1a54678ff50bf27ed9618719b63 {'quay.io/rhceph-dev/rhceph@sha256:d09c53a383ebad581fedbfd2adf155080ebcfd629b17b1cdf5b1ee8433c06120'}
>02:08:08 - MainThread - ocs_ci.ocs.version - INFO  - image quay.io/rhceph-dev/odf4-rook-ceph-rhel9-operator@sha256:a25fbea5d24a9eff136331a9d728038b35610084630a080b73abb536791276df {'quay.io/rhceph-dev/odf4-rook-ceph-rhel9-operator@sha256:a25fbea5d24a9eff136331a9d728038b35610084630a080b73abb536791276df'}
>02:08:08 - MainThread - tests.conftest - INFO  - human readable ocs version info written into /root/ocs_version.2023-04-05T02:08:08.142939
>02:08:08 - MainThread - tests.conftest - INFO  - PagerDuty service is not created because platform from ['openshiftdedicated', 'rosa'] is not used
>02:08:08 - MainThread - ocs_ci.utility.utils - INFO  - testrun_name: OCS4-13-Downstream-OCP4-13-POWERVS-UPI-1AZ-RHCOS-LSO-3M-3W-tier2
>02:08:08 - MainThread - tests.conftest - INFO  - Checking for Ceph Health OK 
>02:08:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc wait --for condition=ready pod -l app=rook-ceph-tools -n openshift-storage --timeout=300s
>02:08:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get pod -l 'app=rook-ceph-tools' -o jsonpath='{.items[0].metadata.name}'
>02:08:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage exec rook-ceph-tools-c8d94fcbd-cx8xl -- ceph health
>02:08:09 - MainThread - ocs_ci.utility.utils - INFO  - Ceph cluster health is HEALTH_OK.
>02:08:09 - MainThread - tests.conftest - INFO  - Ceph health check passed at setup
>02:08:09 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: ['oc', 'login', '-u', 'kubeadmin', '-p', '*****']
>02:08:09 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-monitoring whoami --show-token
>02:08:09 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-monitoring get Route prometheus-k8s -n openshift-monitoring -o yaml
>-------------------------------- live log call ---------------------------------
>02:08:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  --selector=node-role.kubernetes.io/worker -o yaml
>02:08:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc new-project namespace-test-aef02c6fd71b40ba94da79a82
>02:08:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get clusterversion -n openshift-storage -o yaml
>02:08:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc label namespace namespace-test-aef02c6fd71b40ba94da79a82 security.openshift.io/scc.podSecurityLabelSync=false pod-security.kubernetes.io/enforce=baseline pod-security.kubernetes.io/warn=baseline --overwrite
>02:08:10 - MainThread - ocs_ci.ocs.ocp - INFO  - Waiting for a resource(s) of kind storageclass identified by name '' using selector None at column name NAME to reach desired condition ocs-storagecluster-cephfs
>02:08:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get storageclass ocs-storagecluster-cephfs -o yaml
>02:08:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get storageclass ocs-storagecluster-cephfs
>02:08:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get storageclass ocs-storagecluster-cephfs -o yaml
>02:08:10 - MainThread - ocs_ci.ocs.ocp - INFO  - status of ocs-storagecluster-cephfs at NAME reached condition!
>02:08:10 - MainThread - ocs_ci.ocs.resources.ocs - INFO  - Adding PersistentVolumeClaim with name pvc-test-4d92c28aac684d3db4f1251f9df45a8
>02:08:10 - MainThread - ocs_ci.utility.templating - INFO  - apiVersion: v1
>kind: PersistentVolumeClaim
>metadata:
>  name: pvc-test-4d92c28aac684d3db4f1251f9df45a8
>  namespace: namespace-test-aef02c6fd71b40ba94da79a82
>spec:
>  accessModes:
>  - ReadWriteMany
>  resources:
>    requests:
>      storage: 3Gi
>  storageClassName: ocs-storagecluster-cephfs
>
>02:08:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 create -f /tmp/PersistentVolumeClaiml5r6dd3w -o yaml
>02:08:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>02:08:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  --selector=node-role.kubernetes.io/worker -o yaml
>02:08:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  --selector=node-role.kubernetes.io/master -o yaml
>02:08:11 - MainThread - ocs_ci.ocs.ocp - INFO  - Waiting for a resource(s) of kind PersistentVolumeClaim identified by name 'pvc-test-4d92c28aac684d3db4f1251f9df45a8' using selector None at column name STATUS to reach desired condition Bound
>02:08:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get PersistentVolumeClaim pvc-test-4d92c28aac684d3db4f1251f9df45a8 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get PersistentVolumeClaim pvc-test-4d92c28aac684d3db4f1251f9df45a8 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:08:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get PersistentVolumeClaim  -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:12 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pvc-test-4d92c28aac684d3db4f1251f9df45a8 at STATUS reached condition!
>02:08:12 - MainThread - ocs_ci.helpers.helpers - INFO  - PersistentVolumeClaim pvc-test-4d92c28aac684d3db4f1251f9df45a8 reached state Bound
>02:08:12 - MainThread - tests.manage.pv_services.test_pvc_evict_ceph_clients - INFO  - creating first pod on node sao01-worker-1.rdr-abhinew.ibm.com
>02:08:12 - MainThread - ocs_ci.helpers.helpers - INFO  - Creating new Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9 for test
>02:08:12 - MainThread - ocs_ci.ocs.resources.ocs - INFO  - Adding Pod with name pod-test-cephfs-bb958ee6d9b8432991bba3a9
>02:08:12 - MainThread - ocs_ci.utility.templating - INFO  - apiVersion: v1
>kind: Pod
>metadata:
>  name: pod-test-cephfs-bb958ee6d9b8432991bba3a9
>  namespace: namespace-test-aef02c6fd71b40ba94da79a82
>spec:
>  containers:
>  - image: quay.io/ocsci/nginx:latest
>    name: web-server
>    volumeMounts:
>    - mountPath: /var/lib/www/html
>      name: mypvc
>  nodeName: sao01-worker-1.rdr-abhinew.ibm.com
>  volumes:
>  - name: mypvc
>    persistentVolumeClaim:
>      claimName: pvc-test-4d92c28aac684d3db4f1251f9df45a8
>      readOnly: false
>
>02:08:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 create -f /tmp/POD_rsy5k8x2 -o yaml
>02:08:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>02:08:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  --selector=node-role.kubernetes.io/worker -o yaml
>02:08:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  --selector=node-role.kubernetes.io/master -o yaml
>02:08:13 - MainThread - ocs_ci.ocs.ocp - INFO  - Waiting for a resource(s) of kind Pod identified by name 'pod-test-cephfs-bb958ee6d9b8432991bba3a9' using selector None at column name STATUS to reach desired condition Running
>02:08:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:08:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod  -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:13 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-bb958ee6d9b8432991bba3a9 at column STATUS was ContainerCreating, but we were waiting for Running
>02:08:13 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:08:16 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:16 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:08:16 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-bb958ee6d9b8432991bba3a9 at column STATUS was ContainerCreating, but we were waiting for Running
>02:08:16 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:08:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:08:20 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-bb958ee6d9b8432991bba3a9 at column STATUS was ContainerCreating, but we were waiting for Running
>02:08:20 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:08:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:08:23 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-bb958ee6d9b8432991bba3a9 at column STATUS was ContainerCreating, but we were waiting for Running
>02:08:23 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:08:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:08:26 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-bb958ee6d9b8432991bba3a9 at column STATUS was ContainerCreating, but we were waiting for Running
>02:08:26 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:08:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:08:29 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-bb958ee6d9b8432991bba3a9 at column STATUS was ContainerCreating, but we were waiting for Running
>02:08:29 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:08:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:08:32 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-bb958ee6d9b8432991bba3a9 at STATUS reached condition!
>02:08:32 - MainThread - ocs_ci.helpers.helpers - INFO  - Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9 reached state Running
>02:08:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get PersistentVolumeClaim pvc-test-4d92c28aac684d3db4f1251f9df45a8 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get pv pvc-a03d9690-9a87-4cf2-a73f-5b5fd65b196a -o yaml
>02:08:33 - MainThread - tests.manage.pv_services.test_pvc_evict_ceph_clients - INFO  - Evicting ceph-fs clients!
>02:08:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=rook-ceph-tools -o yaml
>02:08:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get Pod  -n openshift-storage --selector=app=rook-ceph-tools
>02:08:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage exec rook-ceph-tools-c8d94fcbd-cx8xl -- bash -c "ceph tell mds.0 client ls"
>02:08:33 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: 2023-04-05T06:08:33.748+0000 7fff6afdf0a0  0 client.79890 ms_handle_reset on v2:10.131.0.61:6800/1500857000
>2023-04-05T06:08:33.768+0000 7fff6afdf0a0  0 client.79896 ms_handle_reset on v2:10.131.0.61:6800/1500857000
>
>02:08:33 - MainThread - tests.manage.pv_services.test_pvc_evict_ceph_clients - INFO  - Client Ids to evict: [65279]
>02:08:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage exec rook-ceph-tools-c8d94fcbd-cx8xl -- bash -c "ceph tell mds.0 client evict id=65279"
>02:08:35 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: 2023-04-05T06:08:34.158+0000 7fff8cf9f0a0  0 client.79908 ms_handle_reset on v2:10.131.0.61:6800/1500857000
>2023-04-05T06:08:35.118+0000 7fff8cf9f0a0  0 client.79914 ms_handle_reset on v2:10.131.0.61:6800/1500857000
>
>02:08:35 - MainThread - tests.manage.pv_services.test_pvc_evict_ceph_clients - INFO  - Clients are evicted successfully!
>02:08:35 - MainThread - tests.manage.pv_services.test_pvc_evict_ceph_clients - INFO  - Running second pod on sao01-worker-1.rdr-abhinew.ibm.com node
>02:08:35 - MainThread - ocs_ci.helpers.helpers - INFO  - Creating new Pod pod-test-cephfs-7587b15391e24385aa2b6df2 for test
>02:08:35 - MainThread - ocs_ci.ocs.resources.ocs - INFO  - Adding Pod with name pod-test-cephfs-7587b15391e24385aa2b6df2
>02:08:35 - MainThread - ocs_ci.utility.templating - INFO  - apiVersion: v1
>kind: Pod
>metadata:
>  name: pod-test-cephfs-7587b15391e24385aa2b6df2
>  namespace: namespace-test-aef02c6fd71b40ba94da79a82
>spec:
>  containers:
>  - image: quay.io/ocsci/nginx:latest
>    name: web-server
>    volumeMounts:
>    - mountPath: /var/lib/www/html
>      name: mypvc
>  nodeName: sao01-worker-1.rdr-abhinew.ibm.com
>  volumes:
>  - name: mypvc
>    persistentVolumeClaim:
>      claimName: pvc-test-4d92c28aac684d3db4f1251f9df45a8
>      readOnly: false
>
>02:08:35 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 create -f /tmp/POD_louafsuk -o yaml
>02:08:35 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:35 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  -o yaml
>02:08:35 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  --selector=node-role.kubernetes.io/worker -o yaml
>02:08:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get Node  --selector=node-role.kubernetes.io/master -o yaml
>02:08:36 - MainThread - ocs_ci.ocs.ocp - INFO  - Waiting for a resource(s) of kind Pod identified by name 'pod-test-cephfs-7587b15391e24385aa2b6df2' using selector None at column name STATUS to reach desired condition Running
>02:08:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:08:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod  -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:36 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:08:36 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:08:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:08:39 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:08:39 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:08:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:08:43 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:08:43 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:08:46 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:46 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:08:46 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:08:46 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:08:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:08:49 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:08:49 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:08:52 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:52 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:08:52 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:08:52 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:08:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:08:55 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:08:55 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:08:58 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:08:58 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:08:59 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:08:59 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:09:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:09:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:09:02 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:09:02 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:09:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:09:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:09:05 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:09:05 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:09:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:09:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:09:08 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:09:08 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:09:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:09:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:09:11 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:09:11 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:09:14 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:09:14 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:09:15 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:09:15 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:09:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:09:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:09:18 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:09:18 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:09:21 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:09:21 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:09:21 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:09:21 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:09:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:09:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:09:24 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:09:24 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:09:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:09:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:09:27 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:09:27 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:09:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:09:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:09:31 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:09:31 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:09:34 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:09:34 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:09:34 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:09:34 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:09:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:09:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:09:37 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:09:37 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:09:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:09:40 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:09:40 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:09:40 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:09:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:09:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:09:43 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:09:43 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:09:46 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:09:46 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:09:47 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:09:47 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:09:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:09:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:09:50 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:09:50 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:09:53 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:09:53 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:09:53 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:09:53 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:09:56 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:09:56 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:09:56 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:09:56 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:09:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:09:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:09:59 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:09:59 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:10:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:10:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:10:03 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:10:03 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:10:06 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:10:06 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:10:06 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:10:06 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:10:09 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:10:09 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:10:09 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:10:09 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:10:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:10:12 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:10:12 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:10:12 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:10:15 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:10:15 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:10:15 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:10:15 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:10:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:10:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:10:19 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:10:19 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:10:22 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:10:22 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:10:22 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:10:22 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:10:25 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:10:25 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:10:25 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:10:25 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:10:28 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:10:28 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:10:28 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:10:28 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:10:31 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:10:31 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:10:31 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:10:31 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:10:34 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:10:34 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:10:35 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:10:35 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:10:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:10:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:10:38 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:10:38 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:10:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:10:41 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:10:41 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:10:41 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:10:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:10:44 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:10:44 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:10:44 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:10:47 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:10:47 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:10:47 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:10:47 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:10:50 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:10:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:10:51 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:10:51 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:10:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:10:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:10:54 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:10:54 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:10:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:10:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:10:57 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:10:57 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:11:00 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:11:00 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:11:00 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:11:00 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:11:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:11:03 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:11:03 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:11:03 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:11:06 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:11:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:11:07 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:11:07 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:11:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:11:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:11:10 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:11:10 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:11:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:11:13 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:11:13 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:11:13 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:11:16 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:11:16 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:11:16 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:11:16 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:11:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:11:19 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:11:19 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:11:19 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:11:22 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:11:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:11:23 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:11:23 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:11:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:11:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:11:26 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:11:26 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:11:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:11:29 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:11:29 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:11:29 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:11:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:11:32 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:11:32 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:11:32 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:11:35 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:11:35 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:11:35 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:11:35 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:11:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:11:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:11:39 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:11:39 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:11:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:11:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:11:42 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:11:42 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:11:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:11:45 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:11:45 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:11:45 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:11:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:11:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:11:48 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:11:48 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:11:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:11:51 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:11:51 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:11:51 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:11:54 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:11:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:11:55 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:11:55 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:11:58 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:11:58 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:11:58 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:11:58 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:12:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:12:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:12:01 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:12:01 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:12:04 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:12:04 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:12:04 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:12:04 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:12:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:12:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:12:07 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:12:07 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:12:10 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:12:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:12:11 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:12:11 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:12:14 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:12:14 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:12:14 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:12:14 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:12:17 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:12:17 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:12:17 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:12:17 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:12:20 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:12:20 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:12:20 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:12:20 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:12:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:12:23 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:12:23 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:12:23 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:12:26 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:12:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:12:27 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:12:27 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:12:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:12:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:12:30 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:12:30 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:12:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:12:33 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:12:33 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:12:33 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:12:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:12:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:12:36 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:12:36 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:12:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:12:39 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:12:39 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:12:39 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:12:42 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:12:43 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:12:43 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:12:43 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:12:46 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:12:46 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:12:46 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:12:46 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:12:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:12:49 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:12:49 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:12:49 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:12:52 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:12:52 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:12:52 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:12:52 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:12:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:12:55 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:12:55 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:12:55 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:12:58 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:12:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:12:59 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:12:59 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:13:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:13:02 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:13:02 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:13:02 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:13:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:13:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:13:05 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:13:05 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:13:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:13:08 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:13:08 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:13:08 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:13:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:13:11 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:13:11 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:13:11 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:13:14 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:13:14 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:13:15 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:13:15 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:13:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:13:18 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:13:18 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:13:18 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:13:21 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:13:21 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:13:21 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:13:21 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:13:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:13:24 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:13:24 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:13:24 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:13:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:13:27 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:13:27 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:13:27 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:13:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:13:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:13:31 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:13:31 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:13:34 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:13:34 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82
>02:13:34 - MainThread - ocs_ci.ocs.ocp - INFO  - status of pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS was ContainerCreating, but we were waiting for Running
>02:13:34 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 3 seconds before next iteration
>02:13:37 - MainThread - ocs_ci.ocs.ocp - ERROR  - timeout expired: Timed out after 300s running get("pod-test-cephfs-7587b15391e24385aa2b6df2", True, None)
>02:13:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 describe Pod pod-test-cephfs-7587b15391e24385aa2b6df2
>02:13:37 - MainThread - ocs_ci.ocs.ocp - WARNING  - Description of the resource(s) we were waiting for:
>Name:             pod-test-cephfs-7587b15391e24385aa2b6df2
>Namespace:        namespace-test-aef02c6fd71b40ba94da79a82
>Priority:         0
>Service Account:  default
>Node:             sao01-worker-1.rdr-abhinew.ibm.com/192.168.0.83
>Start Time:       Wed, 05 Apr 2023 02:08:35 -0400
>Labels:           <none>
>Annotations:      k8s.ovn.org/pod-networks:
>                    {"default":{"ip_addresses":["10.129.3.30/23"],"mac_address":"0a:58:0a:81:03:1e","gateway_ips":["10.129.2.1"],"ip_address":"10.129.3.30/23"...
>                  openshift.io/scc: privileged
>Status:           Pending
>IP:               
>IPs:              <none>
>Containers:
>  web-server:
>    Container ID:   
>    Image:          quay.io/ocsci/nginx:latest
>    Image ID:       
>    Port:           <none>
>    Host Port:      <none>
>    State:          Waiting
>      Reason:       ContainerCreating
>    Ready:          False
>    Restart Count:  0
>    Environment:    <none>
>    Mounts:
>      /var/lib/www/html from mypvc (rw)
>      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-djcbb (ro)
>Conditions:
>  Type              Status
>  Initialized       True 
>  Ready             False 
>  ContainersReady   False 
>  PodScheduled      True 
>Volumes:
>  mypvc:
>    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
>    ClaimName:  pvc-test-4d92c28aac684d3db4f1251f9df45a8
>    ReadOnly:   false
>  kube-api-access-djcbb:
>    Type:                    Projected (a volume that contains injected data from multiple sources)
>    TokenExpirationSeconds:  3607
>    ConfigMapName:           kube-root-ca.crt
>    ConfigMapOptional:       <nil>
>    DownwardAPI:             true
>    ConfigMapName:           openshift-service-ca.crt
>    ConfigMapOptional:       <nil>
>QoS Class:                   BestEffort
>Node-Selectors:              <none>
>Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
>                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
>Events:
>  Type     Reason       Age                  From     Message
>  ----     ------       ----                 ----     -------
>  Warning  FailedMount  56s (x2 over 2m59s)  kubelet  Unable to attach or mount volumes: unmounted volumes=[mypvc], unattached volumes=[mypvc kube-api-access-djcbb]: timed out waiting for the condition
>  Warning  FailedMount  52s (x10 over 5m2s)  kubelet  MountVolume.SetUp failed for volume "pvc-a03d9690-9a87-4cf2-a73f-5b5fd65b196a" : rpc error: code = Internal desc = stat /var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.cephfs.csi.ceph.com/cafb8df0afa1133d486552983ffa9d43376ea73810f04ceb3ea40f58f5a71c1e/globalmount: permission denied
>
>02:13:37 - MainThread - ocs_ci.ocs.ocp - ERROR  - Wait for Pod resource pod-test-cephfs-7587b15391e24385aa2b6df2 at column STATUS to reach desired condition Running failed, last actual status was ContainerCreating
>02:13:37 - MainThread - ocs_ci.helpers.helpers - ERROR  - Pod pod-test-cephfs-7587b15391e24385aa2b6df2 failed to reach Running
>02:13:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:13:37 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 describe Pod pod-test-cephfs-7587b15391e24385aa2b6df2
>02:13:38 - MainThread - ocs_ci.ocs.utils - INFO  - Must gather image: quay.io/rhceph-dev/ocs-must-gather:latest-4.13 will be used.
>02:13:38 - MainThread - ocs_ci.ocs.utils - INFO  - OCS logs will be placed in location /root/logs-ocs-ci/4.13/failed_testcase_ocs_logs_1680674868/test_pvc_evict_ceph_clients[same]_ocs_logs/ocs_must_gather
>02:13:38 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc adm must-gather --image=quay.io/rhceph-dev/ocs-must-gather:latest-4.13 --dest-dir=/root/logs-ocs-ci/4.13/failed_testcase_ocs_logs_1680674868/test_pvc_evict_ceph_clients[same]_ocs_logs/ocs_must_gather
>FAILED
>------------------------------ live log teardown -------------------------------
>02:18:59 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 delete Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9
>02:19:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:19:01 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Error from server (NotFound): pods "pod-test-cephfs-bb958ee6d9b8432991bba3a9" not found
>
>02:19:01 - MainThread - ocs_ci.ocs.ocp - WARNING  - Failed to get resource: pod-test-cephfs-bb958ee6d9b8432991bba3a9 of kind: Pod, selector: None, Error: Error during execution of command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml.
>Error is Error from server (NotFound): pods "pod-test-cephfs-bb958ee6d9b8432991bba3a9" not found
>
>02:19:01 - MainThread - ocs_ci.ocs.ocp - WARNING  - Number of attempts to get resource reached!
>02:19:01 - MainThread - ocs_ci.ocs.ocp - INFO  - Pod pod-test-cephfs-bb958ee6d9b8432991bba3a9 got deleted successfully
>02:19:01 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 delete Pod pod-test-cephfs-7587b15391e24385aa2b6df2
>02:20:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:20:57 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Error from server (NotFound): pods "pod-test-cephfs-7587b15391e24385aa2b6df2" not found
>
>02:20:57 - MainThread - ocs_ci.ocs.ocp - WARNING  - Failed to get resource: pod-test-cephfs-7587b15391e24385aa2b6df2 of kind: Pod, selector: None, Error: Error during execution of command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Pod pod-test-cephfs-7587b15391e24385aa2b6df2 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml.
>Error is Error from server (NotFound): pods "pod-test-cephfs-7587b15391e24385aa2b6df2" not found
>
>02:20:57 - MainThread - ocs_ci.ocs.ocp - WARNING  - Number of attempts to get resource reached!
>02:20:57 - MainThread - ocs_ci.ocs.ocp - INFO  - Pod pod-test-cephfs-7587b15391e24385aa2b6df2 got deleted successfully
>02:20:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get PersistentVolumeClaim pvc-test-4d92c28aac684d3db4f1251f9df45a8 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:20:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get PersistentVolume pvc-a03d9690-9a87-4cf2-a73f-5b5fd65b196a -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:20:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 delete PersistentVolumeClaim pvc-test-4d92c28aac684d3db4f1251f9df45a8
>02:20:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get PersistentVolumeClaim pvc-test-4d92c28aac684d3db4f1251f9df45a8 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:20:57 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Error from server (NotFound): persistentvolumeclaims "pvc-test-4d92c28aac684d3db4f1251f9df45a8" not found
>
>02:20:57 - MainThread - ocs_ci.ocs.ocp - WARNING  - Failed to get resource: pvc-test-4d92c28aac684d3db4f1251f9df45a8 of kind: PersistentVolumeClaim, selector: None, Error: Error during execution of command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get PersistentVolumeClaim pvc-test-4d92c28aac684d3db4f1251f9df45a8 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml.
>Error is Error from server (NotFound): persistentvolumeclaims "pvc-test-4d92c28aac684d3db4f1251f9df45a8" not found
>
>02:20:57 - MainThread - ocs_ci.ocs.ocp - WARNING  - Number of attempts to get resource reached!
>02:20:57 - MainThread - ocs_ci.ocs.ocp - INFO  - PersistentVolumeClaim pvc-test-4d92c28aac684d3db4f1251f9df45a8 got deleted successfully
>02:20:57 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc get PersistentVolume pvc-a03d9690-9a87-4cf2-a73f-5b5fd65b196a -o yaml
>02:20:58 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Error from server (NotFound): persistentvolumes "pvc-a03d9690-9a87-4cf2-a73f-5b5fd65b196a" not found
>
>02:20:58 - MainThread - ocs_ci.ocs.ocp - WARNING  - Failed to get resource: pvc-a03d9690-9a87-4cf2-a73f-5b5fd65b196a of kind: PersistentVolume, selector: None, Error: Error during execution of command: oc get PersistentVolume pvc-a03d9690-9a87-4cf2-a73f-5b5fd65b196a -o yaml.
>Error is Error from server (NotFound): persistentvolumes "pvc-a03d9690-9a87-4cf2-a73f-5b5fd65b196a" not found
>
>02:20:58 - MainThread - ocs_ci.ocs.ocp - WARNING  - Number of attempts to get resource reached!
>02:20:58 - MainThread - ocs_ci.ocs.ocp - INFO  - PersistentVolume pvc-a03d9690-9a87-4cf2-a73f-5b5fd65b196a got deleted successfully
>02:20:58 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Event  -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:20:58 - MainThread - tests.conftest - INFO  - There were 12 events in namespace-test-aef02c6fd71b40ba94da79a82 namespace before it's removal (out of which 2 were of type Warning). For a full dump of this event list, see DEBUG logs.
>02:20:58 - MainThread - ocs_ci.ocs.ocp - INFO  - Switching to project openshift-storage
>02:20:58 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc project openshift-storage
>02:20:58 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 delete Project namespace-test-aef02c6fd71b40ba94da79a82
>02:21:05 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Warning: deleting cluster-scoped resources, not scoped to the provided namespace
>
>02:21:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Project namespace-test-aef02c6fd71b40ba94da79a82 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml
>02:21:05 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Error from server (NotFound): namespaces "namespace-test-aef02c6fd71b40ba94da79a82" not found
>
>02:21:05 - MainThread - ocs_ci.ocs.ocp - WARNING  - Failed to get resource: namespace-test-aef02c6fd71b40ba94da79a82 of kind: Project, selector: None, Error: Error during execution of command: oc -n namespace-test-aef02c6fd71b40ba94da79a82 get Project namespace-test-aef02c6fd71b40ba94da79a82 -n namespace-test-aef02c6fd71b40ba94da79a82 -o yaml.
>Error is Error from server (NotFound): namespaces "namespace-test-aef02c6fd71b40ba94da79a82" not found
>
>02:21:05 - MainThread - ocs_ci.ocs.ocp - WARNING  - Number of attempts to get resource reached!
>02:21:05 - MainThread - ocs_ci.ocs.ocp - INFO  - Project namespace-test-aef02c6fd71b40ba94da79a82 got deleted successfully
>02:21:05 - MainThread - tests.conftest - WARNING  - During test were raised new alerts
>02:21:05 - MainThread - tests.conftest - WARNING  - [{'labels': {'alertname': 'PodStartupStorageOperationsFailing', 'endpoint': 'https-metrics', 'instance': '192.168.0.83:10250', 'job': 'kubelet', 'metrics_path': '/metrics', 'migrated': 'false', 'namespace': 'kube-system', 'node': 'sao01-worker-1.rdr-abhinew.ibm.com', 'operation_name': 'volume_mount', 'service': 'kubelet', 'severity': 'info', 'status': 'fail-unknown', 'volume_plugin': 'kubernetes.io/csi:openshift-storage.cephfs.csi.ceph.com'}, 'annotations': {'description': 'Failing storage operation "volume_mount" of volume plugin kubernetes.io/csi:openshift-storage.cephfs.csi.ceph.com was preventing Pods on node sao01-worker-1.rdr-abhinew.ibm.com\nfrom starting for past 5 minutes.\nPlease investigate Pods that are "ContainerCreating" on the node: "oc get pod --field-selector=spec.nodeName=sao01-worker-1.rdr-abhinew.ibm.com --all-namespaces | grep ContainerCreating".\nEvents of the Pods should contain exact error message: "oc describe pod -n <pod namespace> <pod name>".\n', 'summary': "Pods can't start because volume_mount of volume plugin kubernetes.io/csi:openshift-storage.cephfs.csi.ceph.com is permanently failing on node sao01-worker-1.rdr-abhinew.ibm.com."}, 'state': 'firing', 'activeAt': '2023-04-05T06:09:39.033061875Z', 'value': '1.111111111111111e+01'}, {'labels': {'alertname': 'PodSecurityViolation', 'namespace': 'openshift-kube-apiserver', 'policy_level': 'restricted', 'severity': 'info'}, 'annotations': {'description': 'A workload (pod, deployment, daemonset, ...) was created somewhere in the cluster but it did not match the PodSecurity "restricted" profile defined by its namespace either via the cluster-wide configuration (which triggers on a "restricted" profile violations) or by the namespace local Pod Security labels. Refer to Kubernetes documentation on Pod Security Admission to learn more about these violations.', 'summary': "One or more workloads users created in the cluster don't match their Pod Security profile"}, 'state': 'firing', 'activeAt': '2023-04-04T23:00:31.125647908Z', 'value': '6.006409610444294e+00'}, {'labels': {'alertname': 'CannotRetrieveUpdates', 'namespace': 'openshift-cluster-version', 'severity': 'warning'}, 'annotations': {'description': 'Failure to retrieve updates means that cluster administrators will need to monitor for available updates on their own or risk falling behind on security or other bugfixes. If the failure is expected, you can clear spec.channel in the ClusterVersion object to tell the cluster-version operator to not retrieve updates. Failure reason VersionNotFound .  For more information refer to https://console-openshift-console.apps.rdr-abhinew.ibm.com/settings/cluster/.', 'summary': 'Cluster version operator has not retrieved updates in 23h 36m 12s.'}, 'state': 'firing', 'activeAt': '2023-04-04T23:00:28.020318152Z', 'value': '8.497201999998093e+04'}]
>02:21:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc wait --for condition=ready pod -l app=rook-ceph-tools -n openshift-storage --timeout=300s
>02:21:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage get pod -l 'app=rook-ceph-tools' -o jsonpath='{.items[0].metadata.name}'
>02:21:05 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage exec rook-ceph-tools-c8d94fcbd-cx8xl -- ceph health
>02:21:06 - MainThread - ocs_ci.utility.utils - INFO  - Ceph cluster health is HEALTH_OK.
>02:21:06 - MainThread - tests.conftest - INFO  - Ceph health check passed at teardown
>02:21:06 - MainThread - ocs_ci.framework.pytest_customization.ocscilib - INFO  - Peak rss consumers during tests/manage/pv_services/test_pvc_evict_ceph_clients.py::TestPvcEvictCephClients::test_pvc_evict_ceph_clients[same]:
>+--------+---------------------+---------------------+------------+
>| name   | proc_start          | proc_end            | rss_peak   |
>+========+=====================+=====================+============+
>| run-ci | 2023-04-05 02:08:04 | 2023-04-05 02:21:05 | 302.4M     |
>+--------+---------------------+---------------------+------------+
>| oc     | 2023-04-05 02:08:29 | 2023-04-05 02:21:02 | 80.7M      |
>+--------+---------------------+---------------------+------------+
>| rsync  | 2023-04-05 02:18:58 | 2023-04-05 02:18:58 | 36.9M      |
>+--------+---------------------+---------------------+------------+
>02:21:06 - MainThread - ocs_ci.framework.pytest_customization.ocscilib - INFO  - Peak vms consumers during tests/manage/pv_services/test_pvc_evict_ceph_clients.py::TestPvcEvictCephClients::test_pvc_evict_ceph_clients[same]:
>+--------+---------------------+---------------------+------------+
>| name   | proc_start          | proc_end            | vms_peak   |
>+========+=====================+=====================+============+
>| oc     | 2023-04-05 02:08:29 | 2023-04-05 02:21:02 | 1.3G       |
>+--------+---------------------+---------------------+------------+
>| run-ci | 2023-04-05 02:08:04 | 2023-04-05 02:21:05 | 433.8M     |
>+--------+---------------------+---------------------+------------+
>| rsync  | 2023-04-05 02:18:58 | 2023-04-05 02:18:58 | 160.6M     |
>+--------+---------------------+---------------------+------------+
>02:21:06 - MainThread - ocs_ci.framework.pytest_customization.ocscilib - INFO  - Free RAM gain at the end of the test: 27.9M
>02:21:06 - MainThread - ocs_ci.utility.memory - INFO  - Peak total ram memory consumption: 527.9M at 2023-04-05 02:18:58
>02:21:06 - MainThread - ocs_ci.utility.memory - INFO  - Peak total virtual memory consumption: 3.2G at 2023-04-05 02:18:58
>
>
>=================================== FAILURES ===================================
>__________ TestPvcEvictCephClients.test_pvc_evict_ceph_clients[same] ___________
>
>resource = <ocs_ci.ocs.resources.pod.Pod object at 0x7fff9bde0bb0>
>state = 'Running', timeout = 300
>
>    def wait_for_resource_state(resource, state, timeout=120):
>        """
>        Wait for a resource to get to a given status
>    
>        Args:
>            resource (OCS obj): The resource object
>            state (str): The status to wait for
>            timeout (int): Time in seconds to wait
>    
>        Raises:
>            ResourceWrongStatusException: In case the resource hasn't
>                reached the desired state
>    
>        """
>        if check_cluster_is_compact():
>            timeout = 180
>        if (
>            resource.name == constants.DEFAULT_STORAGECLASS_CEPHFS
>            or resource.name == constants.DEFAULT_STORAGECLASS_RBD
>        ):
>            logger.info("Attempt to default default Secret or StorageClass")
>            return
>        try:
>>           resource.ocp.wait_for_resource(
>                condition=state, resource_name=resource.name, timeout=timeout
>            )
>
>ocs_ci/helpers/helpers.py:116: 
>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>
>self = <ocs_ci.ocs.ocp.OCP object at 0x7fff9d0e9ca0>, condition = 'Running'
>resource_name = 'pod-test-cephfs-7587b15391e24385aa2b6df2', column = 'STATUS'
>selector = None, resource_count = 0, timeout = 300, sleep = 3
>dont_allow_other_resources = False, error_condition = None
>
>    def wait_for_resource(
>        self,
>        condition,
>        resource_name="",
>        column="STATUS",
>        selector=None,
>        resource_count=0,
>        timeout=60,
>        sleep=3,
>        dont_allow_other_resources=False,
>        error_condition=None,
>    ):
>        """
>        Wait for a resource to reach to a desired condition
>    
>        Args:
>            condition (str): The desired state the resource that is sampled
>                from 'oc get <kind> <resource_name>' command
>            resource_name (str): The name of the resource to wait
>                for (e.g.my-pv1)
>            column (str): The name of the column to compare with
>            selector (str): The resource selector to search with.
>                Example: 'app=rook-ceph-mds'
>            resource_count (int): How many resources expected to be
>            timeout (int): Time in seconds to wait
>            sleep (int): Sampling time in seconds
>            dont_allow_other_resources (bool): If True it will not allow other
>                resources in different state. For example you are waiting for 2
>                resources and there are currently 3 (2 in running state,
>                1 in ContainerCreating) the function will continue to next
>                iteration to wait for only 2 resources in running state and no
>                other exists.
>            error_condition (str): State of the resource that is sampled
>                from 'oc get <kind> <resource_name>' command, which makes this
>                method to fail immediately without waiting for a timeout. This
>                is optional and makes sense only when there is a well defined
>                unrecoverable state of the resource(s) which is not expected to
>                be part of a workflow under test, and at the same time, the
>                timeout itself is large.
>    
>        Returns:
>            bool: True in case all resources reached desired condition,
>                False otherwise
>    
>        """
>        if condition == error_condition:
>            # when this fails, this method is used in a wrong way
>            raise ValueError(
>                f"Condition '{condition}' we are waiting for must be different"
>                f" from error condition '{error_condition}'"
>                " which describes unexpected error state."
>            )
>        log.info(
>            (
>                f"Waiting for a resource(s) of kind {self._kind}"
>                f" identified by name '{resource_name}'"
>                f" using selector {selector}"
>                f" at column name {column}"
>                f" to reach desired condition {condition}"
>            )
>        )
>        resource_name = resource_name if resource_name else self.resource_name
>        selector = selector if selector else self.selector
>    
>        # actual status of the resource we are waiting for, setting it to None
>        # now prevents UnboundLocalError raised when waiting timeouts
>        actual_status = None
>    
>        try:
>            for sample in TimeoutSampler(
>                timeout, sleep, self.get, resource_name, True, selector
>            ):
>                # Only 1 resource expected to be returned
>                if resource_name:
>                    retry = int(timeout / sleep if sleep else timeout / 1)
>                    status = self.get_resource(
>                        resource_name,
>                        column,
>                        retry=retry,
>                        wait=sleep,
>                    )
>                    if status == condition:
>                        log.info(
>                            f"status of {resource_name} at {column}"
>                            " reached condition!"
>                        )
>                        return True
>                    log.info(
>                        (
>                            f"status of {resource_name} at column {column} was {status},"
>                            f" but we were waiting for {condition}"
>                        )
>                    )
>                    actual_status = status
>                    if error_condition is not None and status == error_condition:
>                        raise ResourceWrongStatusException(
>                            resource_name,
>                            column=column,
>                            expected=condition,
>                            got=status,
>                        )
>                # More than 1 resources returned
>                elif sample.get("kind") == "List":
>                    in_condition = []
>                    in_condition_len = 0
>                    actual_status = []
>                    sample = sample["items"]
>                    sample_len = len(sample)
>                    for item in sample:
>                        try:
>                            item_name = item.get("metadata").get("name")
>                            status = self.get_resource(item_name, column)
>                            actual_status.append(status)
>                            if status == condition:
>                                in_condition.append(item)
>                                in_condition_len = len(in_condition)
>                            if (
>                                error_condition is not None
>                                and status == error_condition
>                            ):
>                                raise ResourceWrongStatusException(
>                                    item_name,
>                                    column=column,
>                                    expected=condition,
>                                    got=status,
>                                )
>                        except CommandFailed as ex:
>                            log.info(
>                                f"Failed to get status of resource: {item_name} at column {column}, "
>                                f"Error: {ex}"
>                            )
>                        if resource_count:
>                            if in_condition_len == resource_count:
>                                log.info(
>                                    f"{in_condition_len} resources already "
>                                    f"reached condition!"
>                                )
>                                if (
>                                    dont_allow_other_resources
>                                    and sample_len != in_condition_len
>                                ):
>                                    log.info(
>                                        f"There are {sample_len} resources in "
>                                        f"total. Continue to waiting as "
>                                        f"you don't allow other resources!"
>                                    )
>                                    continue
>                                return True
>                        elif len(sample) == len(in_condition):
>                            return True
>                    # preparing logging message with expected number of
>                    # resource items we are waiting for
>                    if resource_count > 0:
>                        exp_num_str = f"all {resource_count}"
>                    else:
>                        exp_num_str = "all"
>                    log.info(
>                        (
>                            f"status of {resource_name} at column {column} - item(s) were {actual_status},"
>                            f" but we were waiting"
>                            f" for {exp_num_str} of them to be {condition}"
>                        )
>                    )
>        except TimeoutExpiredError as ex:
>            log.error(f"timeout expired: {ex}")
>            # run `oc describe` on the resources we were waiting for to provide
>            # evidence so that we can understand what was wrong
>            output = self.describe(resource_name, selector=selector)
>            log.warning(
>                "Description of the resource(s) we were waiting for:\n%s", output
>            )
>            log.error(
>                (
>                    f"Wait for {self._kind} resource {resource_name} at column {column}"
>                    f" to reach desired condition {condition} failed,"
>                    f" last actual status was {actual_status}"
>                )
>            )
>>           raise (ex)
>
>ocs_ci/ocs/ocp.py:731: 
>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>
>self = <ocs_ci.ocs.ocp.OCP object at 0x7fff9d0e9ca0>, condition = 'Running'
>resource_name = 'pod-test-cephfs-7587b15391e24385aa2b6df2', column = 'STATUS'
>selector = None, resource_count = 0, timeout = 300, sleep = 3
>dont_allow_other_resources = False, error_condition = None
>
>    def wait_for_resource(
>        self,
>        condition,
>        resource_name="",
>        column="STATUS",
>        selector=None,
>        resource_count=0,
>        timeout=60,
>        sleep=3,
>        dont_allow_other_resources=False,
>        error_condition=None,
>    ):
>        """
>        Wait for a resource to reach to a desired condition
>    
>        Args:
>            condition (str): The desired state the resource that is sampled
>                from 'oc get <kind> <resource_name>' command
>            resource_name (str): The name of the resource to wait
>                for (e.g.my-pv1)
>            column (str): The name of the column to compare with
>            selector (str): The resource selector to search with.
>                Example: 'app=rook-ceph-mds'
>            resource_count (int): How many resources expected to be
>            timeout (int): Time in seconds to wait
>            sleep (int): Sampling time in seconds
>            dont_allow_other_resources (bool): If True it will not allow other
>                resources in different state. For example you are waiting for 2
>                resources and there are currently 3 (2 in running state,
>                1 in ContainerCreating) the function will continue to next
>                iteration to wait for only 2 resources in running state and no
>                other exists.
>            error_condition (str): State of the resource that is sampled
>                from 'oc get <kind> <resource_name>' command, which makes this
>                method to fail immediately without waiting for a timeout. This
>                is optional and makes sense only when there is a well defined
>                unrecoverable state of the resource(s) which is not expected to
>                be part of a workflow under test, and at the same time, the
>                timeout itself is large.
>    
>        Returns:
>            bool: True in case all resources reached desired condition,
>                False otherwise
>    
>        """
>        if condition == error_condition:
>            # when this fails, this method is used in a wrong way
>            raise ValueError(
>                f"Condition '{condition}' we are waiting for must be different"
>                f" from error condition '{error_condition}'"
>                " which describes unexpected error state."
>            )
>        log.info(
>            (
>                f"Waiting for a resource(s) of kind {self._kind}"
>                f" identified by name '{resource_name}'"
>                f" using selector {selector}"
>                f" at column name {column}"
>                f" to reach desired condition {condition}"
>            )
>        )
>        resource_name = resource_name if resource_name else self.resource_name
>        selector = selector if selector else self.selector
>    
>        # actual status of the resource we are waiting for, setting it to None
>        # now prevents UnboundLocalError raised when waiting timeouts
>        actual_status = None
>    
>        try:
>>           for sample in TimeoutSampler(
>                timeout, sleep, self.get, resource_name, True, selector
>            ):
>
>ocs_ci/ocs/ocp.py:622: 
>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>
>self = <ocs_ci.utility.utils.TimeoutSampler object at 0x7fff9bca0250>
>
>    def __iter__(self):
>        if self.start_time is None:
>            self.start_time = time.time()
>        while True:
>            self.last_sample_time = time.time()
>            if self.timeout <= (self.last_sample_time - self.start_time):
>>               raise self.timeout_exc_cls(*self.timeout_exc_args)
>E               ocs_ci.ocs.exceptions.TimeoutExpiredError: Timed out after 300s running get("pod-test-cephfs-7587b15391e24385aa2b6df2", True, None)
>
>ocs_ci/utility/utils.py:1178: TimeoutExpiredError
>
>During handling of the above exception, another exception occurred:
>
>self = <tests.manage.pv_services.test_pvc_evict_ceph_clients.TestPvcEvictCephClients object at 0x7fff9c12b8e0>
>node = 'same'
>pvc_factory = <function pvc_factory_fixture.<locals>.factory at 0x7fff9d0ef5e0>
>pod_factory = <function pod_factory_fixture.<locals>.factory at 0x7fff9d0ef700>
>
>    @pytest.mark.parametrize(
>        argnames=["node"],
>        argvalues=[
>            pytest.param(
>                "same",
>                marks=[
>                    pytest.mark.bugzilla("1901499"),
>                    pytest.mark.polarion_id("OCS-3985"),
>                ],
>            ),
>            pytest.param(
>                "different",
>                marks=[pytest.mark.polarion_id("OCS-3984")],
>            ),
>        ],
>    )
>    def test_pvc_evict_ceph_clients(self, node, pvc_factory, pod_factory):
>        worker_nodes = get_worker_nodes()
>    
>        # create a RWX PVC
>        pvc_obj = pvc_factory(
>            interface=constants.CEPHFILESYSTEM,
>            access_mode=constants.ACCESS_MODE_RWX,
>            status=constants.STATUS_BOUND,
>        )
>    
>        # create a pod in particular node
>        selected_node = random.choice(worker_nodes)
>        logger.info(f"creating first pod on node {selected_node}")
>        pod_factory(
>            interface=constants.CEPHFILESYSTEM,
>            pvc=pvc_obj,
>            node_name=selected_node,
>            pod_dict_path=constants.NGINX_POD_YAML,
>            status=constants.STATUS_RUNNING,
>        )
>    
>        # fetch the sub-volume path
>        pvc_obj.reload()
>        sub_volume_path = OCP(kind="pv").get(resource_name=pvc_obj.backed_pv)["spec"][
>            "csi"
>        ]["volumeAttributes"]["subvolumePath"]
>    
>        # evict ceph-fs clients
>        logger.info("Evicting ceph-fs clients!")
>        ceph_tools_pod_obj = get_ceph_tools_pod()
>        cmd_ls = "ceph tell mds.0 client ls"
>        output_ls_cmd = json.loads(
>            ceph_tools_pod_obj.exec_sh_cmd_on_pod(command=cmd_ls)
>        )
>        ids = list()
>        for instance in output_ls_cmd:
>            if instance["client_metadata"]["root"] == sub_volume_path:
>                ids.append(instance["id"])
>        logger.info(f"Client Ids to evict: {ids}")
>        try:
>            for id in ids:
>                cmd_evict = f"ceph tell mds.0 client evict id={id}"
>                ceph_tools_pod_obj.exec_sh_cmd_on_pod(command=cmd_evict)
>        except CommandFailed as e:
>            raise ValueError(f"[Error] Client eviction failed: {e}")
>        else:
>            logger.info("Clients are evicted successfully!")
>    
>        # running second pod on same/different node
>        if node == "different":
>            worker_nodes.remove(selected_node)
>            selected_node = random.choice(worker_nodes)
>    
>        logger.info(f"Running second pod on {selected_node} node")
>>       second_pod_obj = pod_factory(
>            interface=constants.CEPHFILESYSTEM,
>            pvc=pvc_obj,
>            node_name=selected_node,
>            pod_dict_path=constants.NGINX_POD_YAML,
>            status=constants.STATUS_RUNNING,
>        )
>
>tests/manage/pv_services/test_pvc_evict_ceph_clients.py:96: 
>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>tests/conftest.py:1098: in factory
>    helpers.wait_for_resource_state(pod_obj, status, timeout=300)
>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>
>resource = <ocs_ci.ocs.resources.pod.Pod object at 0x7fff9bde0bb0>
>state = 'Running', timeout = 300
>
>    def wait_for_resource_state(resource, state, timeout=120):
>        """
>        Wait for a resource to get to a given status
>    
>        Args:
>            resource (OCS obj): The resource object
>            state (str): The status to wait for
>            timeout (int): Time in seconds to wait
>    
>        Raises:
>            ResourceWrongStatusException: In case the resource hasn't
>                reached the desired state
>    
>        """
>        if check_cluster_is_compact():
>            timeout = 180
>        if (
>            resource.name == constants.DEFAULT_STORAGECLASS_CEPHFS
>            or resource.name == constants.DEFAULT_STORAGECLASS_RBD
>        ):
>            logger.info("Attempt to default default Secret or StorageClass")
>            return
>        try:
>            resource.ocp.wait_for_resource(
>                condition=state, resource_name=resource.name, timeout=timeout
>            )
>        except TimeoutExpiredError:
>            logger.error(f"{resource.kind} {resource.name} failed to reach {state}")
>            resource.reload()
>>           raise ResourceWrongStatusException(resource.name, resource.describe())
>E           ocs_ci.ocs.exceptions.ResourceWrongStatusException: Resource pod-test-cephfs-7587b15391e24385aa2b6df2 describe output: Name:             pod-test-cephfs-7587b15391e24385aa2b6df2
>E           Namespace:        namespace-test-aef02c6fd71b40ba94da79a82
>E           Priority:         0
>E           Service Account:  default
>E           Node:             sao01-worker-1.rdr-abhinew.ibm.com/192.168.0.83
>E           Start Time:       Wed, 05 Apr 2023 02:08:35 -0400
>E           Labels:           <none>
>E           Annotations:      k8s.ovn.org/pod-networks:
>E                               {"default":{"ip_addresses":["10.129.3.30/23"],"mac_address":"0a:58:0a:81:03:1e","gateway_ips":["10.129.2.1"],"ip_address":"10.129.3.30/23"...
>E                             openshift.io/scc: privileged
>E           Status:           Pending
>E           IP:               
>E           IPs:              <none>
>E           Containers:
>E             web-server:
>E               Container ID:   
>E               Image:          quay.io/ocsci/nginx:latest
>E               Image ID:       
>E               Port:           <none>
>E               Host Port:      <none>
>E               State:          Waiting
>E                 Reason:       ContainerCreating
>E               Ready:          False
>E               Restart Count:  0
>E               Environment:    <none>
>E               Mounts:
>E                 /var/lib/www/html from mypvc (rw)
>E                 /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-djcbb (ro)
>E           Conditions:
>E             Type              Status
>E             Initialized       True 
>E             Ready             False 
>E             ContainersReady   False 
>E             PodScheduled      True 
>E           Volumes:
>E             mypvc:
>E               Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
>E               ClaimName:  pvc-test-4d92c28aac684d3db4f1251f9df45a8
>E               ReadOnly:   false
>E             kube-api-access-djcbb:
>E               Type:                    Projected (a volume that contains injected data from multiple sources)
>E               TokenExpirationSeconds:  3607
>E               ConfigMapName:           kube-root-ca.crt
>E               ConfigMapOptional:       <nil>
>E               DownwardAPI:             true
>E               ConfigMapName:           openshift-service-ca.crt
>E               ConfigMapOptional:       <nil>
>E           QoS Class:                   BestEffort
>E           Node-Selectors:              <none>
>E           Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
>E                                        node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
>E           Events:
>E             Type     Reason       Age                  From     Message
>E             ----     ------       ----                 ----     -------
>E             Warning  FailedMount  56s (x2 over 2m59s)  kubelet  Unable to attach or mount volumes: unmounted volumes=[mypvc], unattached volumes=[mypvc kube-api-access-djcbb]: timed out waiting for the condition
>E             Warning  FailedMount  52s (x10 over 5m2s)  kubelet  MountVolume.SetUp failed for volume "pvc-a03d9690-9a87-4cf2-a73f-5b5fd65b196a" : rpc error: code = Internal desc = stat /var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.cephfs.csi.ceph.com/cafb8df0afa1133d486552983ffa9d43376ea73810f04ceb3ea40f58f5a71c1e/globalmount: permission denied
>
>ocs_ci/helpers/helpers.py:122: ResourceWrongStatusException
>=============================== warnings summary ===============================
>tests/manage/pv_services/test_pvc_evict_ceph_clients.py::TestPvcEvictCephClients::test_pvc_evict_ceph_clients[same]
>tests/manage/pv_services/test_pvc_evict_ceph_clients.py::TestPvcEvictCephClients::test_pvc_evict_ceph_clients[same]
>  /root/venv/lib64/python3.9/site-packages/urllib3/connectionpool.py:981: InsecureRequestWarning: Unverified HTTPS request is being made to host 'prometheus-k8s-openshift-monitoring.apps.rdr-abhinew.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
>    warnings.warn(
>
>-- Docs: https://docs.pytest.org/en/stable/warnings.html
>=========================== short test summary info ============================
>FAILED tests/manage/pv_services/test_pvc_evict_ceph_clients.py::TestPvcEvictCephClients::test_pvc_evict_ceph_clients[same]
>================== 1 failed, 2 warnings in 784.89s (0:13:04) ===================

Actions: View

Attachments on bug 2187305: 1957836